More Advanced Apps¶
This documentation carries on the Quickstart.
Apps with dependencies¶
When you wrote your first app (jobs.myapp.MyFirstApp) you had to set an app_name on the class. That’s how you reference other apps when setting up dependency management. This is important to note. The name of the Python file or the name of class does not matter.
Diving in, let’s now create two more apps. For simplicity we can continue in the file myapp.py you created. Add this:
class MySecondApp(BaseCronApp): app_name = 'my-second-app' depends_on = ('my-first-app',) # NOTE! def run(self): with open(self.app_name + '.log', 'a') as f: f.write('Second app run at %s\n' % datetime.datetime.now())
Exactly where you add this app doesn’t really matter. Before or after or even in a different file. All the matters is the app_name attribute and the depends_on. It event doesn’t matter which order you place it in your crontabber.ini‘s jobs setting. You can change your crontabber.ini to be like this:
jobs=''' jobs.myapp.MySecondApp|1h jobs.myapp.MyFirstApp|5m '''
crontabber reads the jobs setting but when there’s dependency linking apps, even though it reads the jobs setting from top to bottom, it knows that jobs.myapp.MySecondApp needs to be run after jobs.myapp.MyFirstApp.
Go ahead and try it:
If you now look at the timestamps in my-first-app.log and my-second-app.log are in the correct order according to the dependency linkage rather than the order they are written in the jobs setting.
Another important thing to appreciate is that if a job fails for some reason, i.e. a python exception is raised, it will stop any of the dependening jobs from running. Basically, if job B depends on job A, job B will not run until job A ran without a failure. Basically, crontabber not only makes sure the order is correct, it also guards from running dependents if their “parent” fails.
About the job frequency¶
In the above example note the notation used for the jobs setting. It’s python.module.and.classname|5m or python.module.and.classname|1h.
The frequency is pretty self explanatory. 5m means every 5 minutes and 1h means every hour. The other thing you could use is, for example, 3d meaning every 3 days.
Running at specific times¶
Suppose you have a job that is really intensive and causing a lot of stress in your server. Then you might want to run that “at night” (in quotes because it means different things in different parts of the world) or whenever you have the least load in your server.
The way to specify time is to write it in HH:MM notation on a 24-hour clock. E.g. 22:30.
The way you specify the time is to add it to the jobs second like this example shows:
jobs=''' jobs.myapp.MyBigFatWeddingApp|2d|21:00 ```
But here’s a very important thing to remember. The timezone is that of your PostgreSQL server. Not the timezone of your server. However, when you install PostgreSQL it will take the timezone from your server’s timezone. So if you have a server on the US west coast, the default timezone will be US/Pacific.
However, you can, and it’s a good idea to do, change the timezone of your PostgreSQL server. So if you have set your PostgreSQL server to UTC and the crontabber will adjust these times in UTC time.
Postgres specific apps¶
crontabber provides several class decorators to make use of postgres easier within a crontabber app. These decorators can imbue your app class with the correct configuration to automatically connect with Postgres and handle transactions automatically. The three decorators provide differing levels of automation so that you can choose how much control you want.
This decorator tells crontabber that you want to use postgres by adding to your class two class attributes: self.database_connection_factory and self.database_transaction_executor. When execution reaches your run method, you may use these two attributes to talk to postgres. If you want a connection to Postgres you can grab one from the database_connection_factory and use it as a context manager:
# ... with self.database_connection_factory() as pg_connection: cursor = pg_connection.cursor()
The connection that you get from the factory is a psycopg2 connection, so you have all the resources of that module available for use with your connection. You don’t have to worry about opening or closing the connection, the context manager will do that for you. The connection is open and ready to use when it is handed to you, and is closed when the context ends. You are responsible for transactions within the lifetime of the context.
If you want help with transactions, there is also a the database_transaction_executor at your service. Give it a function that accepts a database connection as its first argument, and it will execute the function within a postgres transaction. If your function ends normally (with or without a return value), the transaction will be automatically committed. If an exception is raised and that exception escapes outside of your function, then the transaction will be automatically rolled back.
@using_postgres() class MyPGApp(BaseCronApp): def execute_lots_of_sql(connection, sql_in_a_list): '''run multiple sql statements in a single transaction''' cursor = connection.cursor() for an_sql_statement in sql_in_a_list: cursor.execute(an_sql_statement) def run(self): sql = [ 'insert into A (a, b, c) values (2, 3, 4)”, 'update A set a=26 where b > 11', 'drop table B' ] self.database_transaction_executor( execute_lots_of_sql, sql_in_a_list )
This decorator is to be used in conjunction with the previous decorator. When using this decorator, your run method must be declared with a database connection as its first argument:
@using_postgres() @with_postgres_connection_as_argument() class MyCrontabberApp(BaseCronApp): app_name = 'postgres-enabled-app' def run(self, connection): # the connection is live and ready to use cursor = connection.cursor() # ...
With this decorator, the database connection is handed to you. You don’t have to get it yourself. You don’t have to worry about closing the connection, it will be closed for you when your ‘run’ function ends. However, you are still responsible for your own transactions: you must explicitly use ‘commit’ or ‘rollback’. If you do not ‘commit’ your changes, they will be lost when the connection gets closed at the end of your function.
You still have the transaction manager available if you want to use it. Note, however, that it will acquire its own database connection and not use the one that was passed into your run function. Don’t deadlock yourself.
This decorator gives you the most automation. It considers your entire run function to be a single postgres transaction. You’re handed a connection through the parameters to your run function. You use that connection to accomplish database stuff. If your run function exits normally, the ‘commit’ will happen automatically. If your run function exits with a Exception being raised, the connection will be rolled back automatically.
@using_postgres() @as_single_postgres_transaction() class MyCrontabberApp(BaseCronApp): app_name = 'postgres-enabled-app' def run(self, connection): # the connection is live and ready to use cursor = connection.cursor() cusor.execute('insert into A (a, b, c) values (11, 22, 33)') if bad_situation_detected(): raise GetMeOutOfHereError()
In this example, connections are as automatic as we can make them. If the exception is raised, the insert will be rolled back. If the exception is not raised and the ‘run’ function exits normally, the insert will be committed.
crontabber is all Python but some of the tasks might be something other than Python. For example, you might want to run rm /var/logs/oldjunk.log or something more advanced.
What you do then is use the with_subprocess helper. When you use this helper on your application class, you can use self.run_process() and it will return a tuple of exit code, stdout, stderr. This example shows how to use it:
from crontabber.base import BaseCronApp from crontabber.mixins import with_subprocess @with_subprocess class MyFirstCommandlineApp(BaseCronApp): app_name = 'my-first-commandline-app' def run(self): command = 'rm -f /var/logs/oldjunk.log' exit_code, stdout, stderr = self.run_process(command) if exit_code != 0: self.config.logger.error( 'Failed to execute %r' % command, ) raise Exception(stderr)