Getting Started Hacking on Koji¶
This page gives an overview of the Koji code and then describes whatneeds to change if you want to add a new type of task. A new task couldbe for a new content type, or assembling the results of multiple buildstogether, or something else that helps your workflow. New contributorsto Koji should leave this page knowing where to begin and have enoughunderstanding of Koji’s architecture to be able to estimate how muchwork is still ahead of them.
Task Flow¶
A task starts with a user submitting it with the Koji client, which is acommand line interface. This contacts the hub, an apache-based serverapplication. It leaves a row in the database that represents a “free”task, one that has not been assigned to a builder. Periodically, thebuilders asynchronously ping the hub asking if there are any tasksavailable, and at some point one will be given the new task. The hubmarks this in the database, and the builder begins executing the task (abuild).
Upon completion, the builder uploads the results to the hub, includinglogs, binaries, environment information, and whatever else the taskhandler for the build dictated. The hub moves the results to a permanentshared storage solution, and marks the task as completed (or failed).During this whole time, the webUI can be used to check up on progress.So the flow of work is:
Client -> Hub -> Builder -> Hub
If you wanted to add a new build type or task that was tightlyintegrated in Koji’s data model, you would need to modify the CLI, Hub,Builder, and WebUI at a minimum. Alternatively, you could do this with aplugin, which is far simpler but less flexible.
Component Overview¶
Koji is comprised of several components, this section goes into detailsfor each one, and what you potentially may need to change. Everycomponent is written in Python, so you will need to know that languagebeyond a beginner level.
Koji-client¶
koji-client is a command line interface that provides many hooks intoKoji. It allows the user to query much of the data as well as performactions such as adding users and initiating build requests.
Option Handling¶
The code is in cli/koji. It uses OptionParsers extensively withinterspersed arguments disabled. That means these two commands are notinterpreted the same:
$ koji -u admin -p password tag-build some-tag --force some-build$ koji tag-build -u admin -p password some-tag --force some-build
The second one will generate an error, because -u and -p are not optionsfor tag-build, they must show up before that because they are globaloptions that can be used with any subcommand. There will be twoOptionParsers used with each command. The first is used to pick uparguments to koji itself, and the second for the subcommandspecified. When the first one executes (see get_options()) it willfigure out the subcommand and come up with a function name based on it.
The convention is to prepend the word handle_ before it, and changeall hyphens to underscores. If a command does not require an accountwith Koji, the function handle will prepended with anon_handle_instead. The code will dynamically call the derived function handlewhich is where the second OptionParser is used to parse theremaining options. To have your code log into Koji (you’re writing ahandle_ function), use the activate_session function. All functionsignatures in the client code will get a session object, which is yourinterface to the hub.
Profiles¶
It is possible to run the Koji client with different configurationprofiles so that you can interact with multiple Koji instances easily.The --profile option to the Koji command itself enables this. Youshould have a ~/.koji/config already, if not just copy from/etc/koji.conf to get a start. The profile command accepts anargument that matches a section in that config file. So if your configfile had this:
[Fedora]authtype = sslserver = https://koji.fedoraproject.org/kojihubtopdir = /mnt/kojiweburl = https://koji.fedoraproject.org/koji#pkgurl = https://koji.fedoraproject.org/packagescert = ~/.fedora.certca = ~/.fedora-upload-ca.certserverca = ~/.fedora-server-ca.cert[MyKoji]server = https://koji.mydomain.com/kojihubauthtype = kerberostopdir = /mnt/kojiweburl = https://koji.mydomain.com/kojitopurl = https://download.mydomain.com/kojifiles
you could pass Fedora or MyKoji to –profile.
Creating Tasks¶
Once options are processed and understood, a task needs to be created onthe hub so that a builder can come along and take it. This isaccomplished with the makeTask method (defined on the Hub, so callit on the session object). The name of the task should match thename given to the task handler in the builder, which is explained lateron.
Be sure to process the channel, priority, background, and watch/nowatchparameters too, which should be available to most new tasks. They’ll beburied in the first argument to your handler function, which capturesthe options passed to the base Koji command.
If the client needs to make locally-available artifacts (config files,sources, kickstarts) accessible to the builder, it must be uploaded tothe hub. This is the case with uploading SRPMs or kickstarts. You caneasily upload this content with the session.uploadWrapper method.You can create progress bars as necessary with this snippet:
if _running_in_bg() or task_opts.noprogress: callback = Noneelse: callback = _progress_callbackserverdir = unique_path('cli-image') # create a unique path on the hubsession.uploadWrapper(somefile, serverdir, callback=callback)
Task Arguments¶
If you define a new task for Koji, you’ll want the task submissionoutput to have the options ordered usefully. This output isautomatically generated, but sometimes it does not capture the moreimportant arguments you want displayed.
Created task 10001810Watching tasks (this may be safely interrupted)...10001810 thing (noarch): free10001810 thing (noarch): free -> closed 0 free 0 open 1 done 0 failed10001810 thing (noarch) completed successfully
In this (fake) example, you can see that “noarch” is the only optionbeing displayed, but maybe you want something more than just the taskarchitecture displayed, like some other options that were passed in. Youcan fix this behavior in koji/__init__.py in the _taskLabelfunction. Here you can define the string(s) to display when Kojireceives status on a task. That is the return value.
Koji-Hub¶
koji-hub is the center of all Koji operations. It is an XML-RPC serverrunning under mod_wsgi in Apache. koji-hub is passive in that it onlyreceives XML-RPC calls and relies upon the build daemons and othercomponents to initiate communication. koji-hub is the only componentthat has direct access to the database and is one of the two componentsthat have write access to the file system. If you want to make changesto the webUI (new pages or themes), you are looking in the wrongsection, there is a separate component for that.
Implementation Details¶
The hub/kojihub.py file is where the server-side code lives. If youneed to fix any server problems or want to add any new tasks, you willneed to modify this file. Changes to the database schema will almostcertainly require code changes too. This file gets deployed to/usr/share/koji-hub/kojihub.py, whenever you make changes to thatremember to restart httpd. Also there are cases where httpd looksfor an existing .pyc file and takes it as-is, instead of re-compiling itwhen the code is changed.
In the code there are two large classes: RootExports andHostExports. RootExports exposes methods using XMLRPC for any clientthat connects to the server. The Koji CLI makes use of this quite a bit.If you want to expose a new API to any remote system, add your codehere. The HostExports class does the same thing except it will ensurethe requests are only coming from builders. Attempting to use an APIexposed here with the CLI will fail. If your work requires the buildersto call a new API, you should implement it here. Any other functiondefined in this file is inaccessible by remote hosts. It is generally agood practice to have the exposed APIs do very little work, and pass offcontrol to internal functions to do the heavy lifting.
Database Interactions¶
Database interactions are done with raw query strings, not with any kindof modern ORM. Consider using context objects from the Koji contextslibrary for thread-safe interactions. The database schema is captured inthe docs directory in the root of a git clone. A visualization ofthe schema is not available at the time of this writing.
If you plan to introduce schema changes, please update bothschema.sql and provide a migration script if necessary.
Troubleshooting¶
The hub runs in an Apache service, so you will need to look in Apachelogs for error messages if you are encountering 500 errors or theservice is failing to start. Specifically you want to check in:
/var/log/httpd/error_log
/var/log/httpd/ssl_error_log
If you need more specific tracebacks and debugging data, considerchanging the debugging setting in /etc/koji-hub/hub.conf. Be advisedthe hub is very verbose with this setting on, your logs will take upgigabytes of space within several days.
Kojid¶
kojid is the build daemon that runs on each of the build machines. Itsprimary responsibility is polling for incoming build requests andhandling them accordingly. Essentially kojid asks koji-hub for work.Koji also has support for tasks other than building. Creating installimages is one example. kojid is responsible for handling these tasks aswell. kojid uses mock for building. It also creates a fresh buildrootfor every build. kojid is written in Python and communicates withkoji-hub via XML-RPC.
Implementation Details¶
The daemon runs as a service on a host that is traditionally not thesame as the hub or webUI. This is a good security practice because theservice runs as root, and executes untrusted code to produce builds on aregular basis. Keeping the Hub separate limits the damage a maliciouspackage can do to the build system as a whole. For the same reason, thefilesystem that the hub keeps built software on should be mountedRead-Only on the build host. It should call APIs on the hub that areexposed through the HostExports class in the hub code. Whenever thebuilder accepts a task, it forks a process to carry out the build.
An initscript/unit-file is available for kojid, so it can be stopped andstarted like a normal service. Remember to do this when you deploychanges!
TaskHandlers¶
All tasks in kojid have a TaskHandler class that defines what to dowhen the task is picked up from the hub. The base class is defined inkoji/tasks.py where a lot of useful utility methods are available.An example is uploadFile, which is used to upload logs and builtbinaries from a completed build to the hub since the shared filesystemis read only.
The daemon code lives in builder/kojid, which is deployed to/usr/sbin/kojid. In there you’ll notice that each task handler class hasa Methods member and _taskWeight member. These must be defined,and the former is used to match the name of a waiting task (on the hub)with the task handler code to execute. Each task handler object musthave a handler method defined, which is the entry point for theforked process when a builder accepts a task.
Tasks can have subtasks, which is a typical model when a build can berun on multiple architectures. In this case, developers should write 2task handlers: one handles the build for exact one architecture, and onethat assembles the results of those tasks into a single build, and sendsstatus information to the hub. You can think of the latter handler asthe parent task.
All task handler objects have a session object defined, which is theinterface to use for communications with the hub. So, parent tasksshould kick off child tasks using the session object’s subtask method(which is part of HostExports). It should then call self.wait withall=True to wait for the results of the child tasks.
Here’s a stub of what a new build task might look like:
class BuildThingTask(BaseTaskHandler): Methods = ['thing'] _taskWeight = 0.5 def handler(self, a, b, arches, options): subtasks = {} for arch in arches: subtasks[arch] = session.host.subtask(method='thingArch', a, b, arch) results = self.wait(subtasks.values(), all=True) # parse results and put rows in database # put files in their final resting place return 'Build successful'class BuildThingArchTask(BaseTaskHandler): Methods = ['thingArch'] _taskWeight = 2.0 def handler(self, a, b, arch): # do the build, capture results in a variable self.uploadFile('/path/to/some/log') self.uploadFile('/path/to/binary/file') return result
Source Control Managers¶
If you your build needs to check out code from a Source Control Manager(SCM) such as git or subversion, you can use SCM objects defined inkoji/daemon.py. They take a specially formed URL as an argument tothe constructor. Here’s an example use. The second line is important, itmakes sure the SCM is in the whitelist of SCMs allowed in/etc/kojid/kojid.conf.
scm = SCM(url)scm.assert_allowed(self.options.allowed_scms)directory = scm.checkout('/checkout/path', session, uploaddir, logfile)
Checking out takes 4 arguments: where to checkout, a session object(which is how authentication is handled), a directory to upload the logto, and a string representing the log file name. Using this method Kojiwill checkout (or clone) a remote repository and upload a log of thestandard output to the task results.
Build Root Objects¶
It is encouraged to build software in mock chroots if appropriate. Thatway Koji can easily track precise details about the environment in whichthe build was executed. In builder/kojid a BuildRoot class isdefined, which provides an interface to execute mock commands. Here’s anexample of their use:
broot = BuildRoot(self.session, self.options, build_tag, arch, self.id)
A session object, task options, and a build tag should be passed inas-is. You should also specify the architecture and the task ID. If youever need to pass in specialized options to mock, look in theImageTask.makeImgBuildRoot method to see how they are defined and passedin to the BuildRoot constructor.
Troubleshooting¶
The daemon writes a log file to /var/log/kojid.log. Debugging outputcan be turned on in /etc/kojid/kojid.conf.
Koji-Web¶
koji-web is a set of scripts that run in mod_wsgi and use the Cheetahtemplating engine to provide a web interface to Koji. It acts as aclient to koji-hub providing a visual interface to perform a limitedamount of administration. koji-web exposes a lot of information and alsoprovides a means for certain operations, such as cancelling builds.
The web pages are derived from Cheetah templates, the syntax of whichyou can read up onhere. Thesetemplates are the chtml files sitting in www/kojiweb. You’llnotice quickly that these templates are referencing variables, but wheredo they come from?
The www/kojiweb/index.py file provides them. There are severalfunctions named after the templates they support, and in each one adictionary called values is populated. This is how data is gatheredabout the task, build, archive, or whatever the page is about. Take yourtime with taskinfo.chtml in particular, as the conditionals therehave gotten quite long. If you are adding a new task to Koji, you willneed to extend this at a minimum. A new type of build task would requirethis, and possibly another that is specific to viewing the archivedinformation about the build. (taskinfo vs. buildinfo)
If your web page needs to display the contents of a list or dictionary,use the $printMap function to help with that. It is often sensibleto define a function that easily prints options and values in adictionary. An example of this is in taskinfo.chtml.
#def printOpts($opts) #if $opts <strong>Options:</strong><br/> $printMap($opts, ' ') #end if#end def
Finally, if you need to expand the drop-down menus of “method” typeswhen searching for tasks in the WebUI, you will need to add them to the_TASKS list in www/kojiweb/index.py. Add values whereappropriate to _TOPLEVEL_TASKS and _PARENT_TASKS as well so thatparent-child relationships show up correctly too.
Remember whenever you update a template or index.py, you will need todeploy and restart apache/httpd!
Troubleshooting¶
Like the hub, this component is backed by apache, so you should followthe same techniques for debugging Koji-Web asKoji-Hub.
Kojira¶
kojira is a daemon that keeps the build root repodata updated. It isresponsible for removing redundant build roots and cleaning up after abuild request is completed.
Building and Deploying Changes¶
The root of the git clone for Koji code contains a Makefile that hasa few targets to make building and deployment a little easier. Amongthem are:
tarball: create a bz2 tarball that could be consumed in an rpm build
rpm: create Koji rpms. The NVRs will be defined by the spec file,which is also in the same directory. The results will appear in a
noarchdirectory.test-rpm: like rpm, but append the Release field with a date and timestamp for easy upgrade-deployment
Plugins¶
This section is copied from the docs/Writing_a_plugin.md file.
Koji supports different types of plugins, three of which are capturedhere. Depending on what you are trying to do, there are different waysto write a Koji plugin.
Koji Builder Plugins¶
Koji can do several things, for example build RPMs, or live CDs. Thoseare types of tasks which Koji knows about. If you need to do somethingwhich Koji does not know yet how to do, you could create a Koji Builderplugin. Such a plugin would minimally look like this:
from koji.tasks import BaseTaskHandlerclass MyTask(BaseTaskHandler): Methods = ['mytask'] _taskWeight = 2.0def handler(self, arg1, arg2, kwarg1=None): self.logger.debug("Running my task...") # Here is where you actually do something
A few explanations on what goes on here:
Your task needs to inherit from `koji.tasks.BaseTaskHandler`
Your task must have a `Methods` attribute, which is a list of themethod names your task can handle.
You can specify the weight of your task with the `_taskWeight`attribute. The more intensive (CPU, IO, …) your task is, the higherthis number should be.
The task object has a
loggerattribute, which is a Python loggerwith the usual `debug`, `info`, `warning` and `error`methods. The messages you send with it will end up in the KojiBuilder log.Your task must have a `handler()` method. That is the method Kojiwill call to run your task. It is the method that should actually dowhat you need. It can have as many positional and named arguments asyou want.
Save your plugin as e.g mytask.py, then install it in the KojiBuilder plugins folder: /usr/lib/koji-builder-plugins/. Finally,edit the Koji Builder config file, /etc/kojid/kojid.conf:
# A space-separated list of plugins to enableplugins = mytask
Restart the Koji Builder service, and your plugin will be enabled. Youcan try running a task from your new task type with the command-line:$ koji make-task mytask arg1 arg2 kwarg1
Hub Plugins¶
Koji clients talk to the Koji Hub via an XMLRPC API. It is sometimesdesirable to add to that API, so that clients can request things Kojidoes not expose right now. Such a plugin would minimally look like this:
def mymethod(arg1, arg2, kwarg1=None): # Here is where you actually do something mymethod.exported = True
What’s happening?
Your plugin is just a method, with whatever positional and/or namedarguments you need.
You must export your method by setting its
exportedattribute toTrueThe
context.session.assertPerm()is how you ensure that thecorrect permissions are available.
Save your plugin as e.g `mymethod.py`, then install it in the Koji Hubplugins folder, which is /usr/lib/koji-hub-plugins/
Finally, edit the Koji Hub config file, /etc/koji-hub/hub.conf:
# A space-separated list of plugins to enablePlugins = mymethod
Restart the Koji Hub service, and your plugin will be enabled. You cantry calling the new XMLRPC API with the Python client library:
>>> import koji>>> session = koji.ClientSession("http://koji/example.org/kojihub")>>> session.mymethod(arg1, arg2, kwarg1='some value')
If you want your new XMLRPC API to require specific permissions from theuser, all you need to do is add the following to your method:
from koji.context import contextdef mymethod(arg1, arg2, kwarg1=None): context.session.assertPerm("admin") # Here is where you actually do something mymethod.exported = True
In the example above, Koji will ensure that the user is anadministrator. You could of course create your own permission, and checkfor that.
Event Plugin¶
You might want to run something automatically when something elsehappens in Koji. A typical example is to automatically sign a packageright after a build finished. Another would be to send a notification toa message bus after any kind of event.
This can be achieved with a plugin too, which would look minimally asfollows:
from koji.plugin import callback@callback('preTag', 'postTag')def mycallback(cbtype, tag, build, user, force=False): # Here is where you actually do something
So what is this doing?
The
@callbackdecorator allows you to declare which events shouldtrigger your function. You can pass as many as you want. For a listof supported events, seekoji/plugins.py.The arguments of the function depend on the event you subscribed to.As a result, you need to know how it will be called by Koji. Youprobably should use
*kwargsto be safe. You can see how callbacksare called in thehub/kojihub.pyfile, search for calls of therun_callbacksfunction.
Save your plugin as e.g mycallback.py, then install it in the KojiHub plugins folder: /usr/lib/koji-hub-plugins
Finally, edit the Koji Hub config file, /etc/koji-hub/hub.conf:
# A space-separated list of plugins to enablePlugins = mycallback
Restart the Koji Hub service, and your plugin will be enabled. You cantry triggering your callback plugin with the command-line. For example,if you registered a callback for the postTag event, try tagging abuild: $ koji tag-build mytag mypkg-1.0-1
Submitting Changes¶
To submit code changes for Koji, please file a pull request in Pagure.
https://pagure.io/koji/pull-requests
Here are some guidelines on producing preferable pull requests.
Each request should be a coherent whole, e.g. a single feature or bug fix.Please do not bundle a series of unrelated changes into a single PR
Pull requests in Pagure come from a branch in your personal fork of Koji(either in Pagure or a remote git repo). Please use an appropriately namedbranch for this. Do not use the master branch of your fork. Also, pleasebe aware that Pagure will automatically update the pull request if youmodify the source branch
Your branch should be based against the current HEAD of the target branch
Please adhere to PEP8.While much of the older code in Koji does not, we try to stick to itwith new code
Code which is imported into CLI or needed for stand-alone API calls mustrun in both 2.6+ and 3.x python versions. We use the python-six libraryfor compatibility. The affected files are:
cli/*koji/__init__.pykoji/auth.pykoji/tasks.pykoji/util.pytests/test_lib/*tests/test_cli/*
Check, that unit tests are not broken. Simply run
make testin maindirectory of your branch. For python3 compatible-code we have alsomaketest3target.
Note that the core development team for Koji is small, so it may take a fewdays for someone to reply to your request.
Partial work¶
Pull requests are for changes that are complete and ready for inclusion, butsometimes you have partial work that you may want feedback on. Please don’tsubmit a PR before your code is complete.
The preferred way to request early feedback is to push your changes to a yourown koji fork and then send an email tokoji-devel AT lists.fedorahosted.orgrequesting review. This approach is one step short of a PR, making it easy toupgrade to a PR once the changes are ready.
Unit Tests¶
Koji comes with a small test suite, that you should always run when makingchanges to the code. To do so, just run make test in your terminal.
You will need to install the following packages to actually run the tests.
findutils
pyOpenSSL
python-coverage
python-krbV
python-mock
python-psycopg2
python-requests
python-qpid-proton
Please note that it is currently not supported to use virtualenv when hackingon Koji.
Unit tests are run automatically for any commit in master branch. We useFedora’s jenkins instance for that. Details are given here: Unit testsin Fedora’s Jenkins.
