December 2, 2011

Celery init scripts for Gentoo

by orzel
Categories: Admin, Django, Gentoo
Tags: No Tags
Comments: 5 Comments

I’m using django-celery on a project. The only difficult part was that gentoo ebuilds would not provide init scripts. It might be that some ebuilds in some obscure overlay provides those, but this was far too far away from the mainstream portage tree for me.

Yes, the documentation about celery has some scripts using supervisord, but I still like the init.d kind of scripts, so here they are.

The first one is /etc/init.d/celeryd, used for every worker.

#!/sbin/runscript
# Copyright 2011 Sylphide Consulting / Thomas Capricelli

# this file should be installed as /etc/init.d/celeryd

# doc:
# http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=4#doc_chap4

# configuration is taken from /etc/conf.d/celeryd
# You NEED to configure CELERYD_APP_DIR
# You MAY configure CELERYD_USER, CELERYD_GROUP, CELERYD_CONCURRENCY

depend() {
need net

}

checkconfig() {
if [ -z "${CELERYD_APP_DIR}" ] ; then
eerror "Please configure /etc/conf.d/celeryd"
return 1
fi
}

start() {
concurrency=${CELERYD_CONCURRENCY:-1}
options="--workdir=${CELERYD_APP_DIR} --pidfile=/var/run/celeryd.pid --logfile=/var/log/celeryd.log --concurrency=${concurrency} --events"
if [ -n "${CELERYD_USER}" ] ; then
options="${options} --uid=${CELERYD_USER}"
fi
if [ -n "${CELERYD_GROUP}" ] ; then
options="${options} --gid=${CELERYD_GROUP}"
fi
ebegin "Starting celeryd"
start-stop-daemon \
--chdir ${CELERYD_APP_DIR} \
--start \
--pidfile /var/run/celeryd.pid \
--exec ${CELERYD_APP_DIR}/manage.py -- \
celeryd_detach ${options}
eend $?
}

stop() {
ebegin "Stopping celeryd"
start-stop-daemon --stop --pidfile=/var/run/celeryd.pid
eend $?
}

The second one is /etc/init.d/celerybeat, for the scheduler. I’m using redis, but if you dont, just comment out the use redis line.

#!/sbin/runscript
# Copyright 2011 Sylphide Consulting / Thomas Capricelli

# this file should be installed as /etc/init.d/celerybeat

# doc:
# http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?part=2&chap=4#doc_chap4

# configuration is taken from /etc/conf.d/celerybeat
# You NEED to configure CELERYBEAT_APP_DIR
# You MAY configure CELERYBEAT_USER, CELERYBEAT_GROUP

depend() {
need net
use redis
}

checkconfig() {
if [ -z "${CELERYBEAT_APP_DIR}" ] ; then
eerror "Please configure /etc/conf.d/celerybeat"
return 1
fi
}

start() {
options="--workdir=${CELERYBEAT_APP_DIR} --pidfile=/var/run/celerybeat.pid --logfile=/var/log/celerybeat.log --detach"
if [ -n "${CELERYBEAT_USER}" ] ; then
options="${options} --uid=${CELERYBEAT_USER}"
fi
if [ -n "${CELERYBEAT_GROUP}" ] ; then
options="${options} --gid=${CELERYBEAT_GROUP}"
fi
ebegin "Starting celerybeat"
start-stop-daemon \
--chdir ${CELERYBEAT_APP_DIR} \
--start \
--pidfile /var/run/celerybeat.pid \
--exec ${CELERYBEAT_APP_DIR}/manage.py -- \
celerybeat ${options}
eend $?
}

stop() {
ebegin "Stopping celerybeat"
start-stop-daemon --stop --pidfile=/var/run/celerybeat.pid
eend $?
}

 

edit (march 2nd, 2013) : changed ‘after’ to ‘use’, as suggested in comment. Thanks Alice.


5 Comments »

  1. Hi,
    You will probably be interested by my overlay http://git.iksaif.net/?p=portage.git, I recently rewrote some ebuilds for celery, django-celery and their dependencies. I also rewrote and init script for gentoo, it allow to launch celerydm celerybeat and/or celeryev.

  2. Hey there, I have the not-so-obscure but certainly outdated overlay on gentoo.org with the celery (and related) ebuilds. I didn’t know anyone was using them, since no one ever posted to the bug or emailed me stating as much, which I had requested. I have a bunch of updates in a private overlay I guess I should look at merging into my developer overlay along with Corentin’s work and yours.

    In the case that you are interested in the celery ecosystem being included in the actual portage tree, please let me know via email or post to one of the bugs and I can get this pushed in. I kept it in my overlay due to what I perceived as a lack of interest from users. That said, I have been processing millions of jobs for the past couple years with a rabbitmq+celery cluster, its awesome.

    https://bugs.gentoo.org/342041 celeryd
    https://bugs.gentoo.org/342043 django-celery

    Also, for the record, the ebuilds in my official gentoo developer overlay do provide init scripts. http://git.overlays.gentoo.org/gitweb/?p=dev/quantumsummers.git

    You may want to update your post to let your readers know about the above.

    Also, I hate your spam thing here, its full of fail with JS disabled

  3. I approve all non-spam comments, readers will know about your stuff. I’ll contact you privately about merging scripts.

    I agree celery is awesome! 🙂 I’m less convinced by rabbitmq, but redis is ok for me.

    I’m not fond of my antispam plugins either, but without both of them, i’m getting several spams a day and it was too painful to moderate. If anyone has some good (and free) wordpress antispam plugin, i’m all interested.

  4. Alice Bevan-McGregor says:

    Howdy! Great top hit when Googling for gentoo celery, but I have one recommendation: instead of “after X” (which is mostly useful when injecting dependencies into existing init.d scripts via their conf.d companion) write “use X”. It’s like “needs X” but as a non-hard dependency, while still shutting down things that “use” the package if you restart the dependency.

    Basically if you restart your queue you want the workers to be shut down, too, or there might be Badness™.

  5. @Alice : You convinced me, and i’ve updated the post accordingly. Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

CAPTCHA image
*