CPython Extension Modules: An Introduction

Create, test and package CPython Extension modules.

Photo by David Clode on Unsplash

There are several ways to create Extension modules for Python. One of them is to use the Python API. Using Python API It is possible to code new built-in extensions for python in C or C++. Python API defines a set of functions, macros, and variables that provide access to most aspects of the Python run-time system. CPython is the default implementation of the Python language. CPython compiles the python source code into intermediate bytecode, which is executed by the CPython virtual machine. In this article, we’ll learn to create a simple Python extension module with CPython.

What this article covers

• How to invoke C function from python.
• How to pass arguments from python to C and return values from C to Python.
• How to package and test the extension module.

Why CPython?

• CPython allows you to implement new built-in object types.
• CPython allows you to call C library functions and system calls.
• CPython Extensin modules can run much faster than pure python modules.

Implement python interface in C

We’ll be writing a wrapper for the C library function `copysign()`. It returns the floating-point value with the magnitude of x and the sign of y, where x and y are the arguments of the function. more info about `copysign()` function is available here.

Let's start by adding <python.h> header file. All of the tools required to use Python API are bundled in this header file.

We need an init function for our module.

PyMODINIT_FUNC
PyInit_copy_sign(void) {
    return PyModule_Create(&copy_sign);
}

• When a python module is imported it’ll call the PyInit_name functions where the name is the name of the module.
• PyMODINIT_FUNC declares the function as PyObject * return type and declares any special linkage required for the platform.
• PyModule_Create() will return a new module object of type PyObject *.

It's time to define a structure to store the information about our module.

static struct PyModuleDef copy_sign = {
    PyModuleDef_HEAD_INIT,
    "copy_sign",
    "A Python interface for copy_sign function in C",
    -1,
    copy_sign_methods,
};

• PyModuleDef is the struct that holds information about our module.
• There is a total of 9 elements in this struct, but we don't need them all. we will be defining
• PyModuleDef_Base m_base - Always initialize this member to PyModuleDef_HEAD_INIT
• const char *m_name - Name for the new module.
• const char *m_doc - Docstring for the module.
• Py_ssize_t m_size - This memory area is allocated based on m_size on module creation and freed when the module object is deallocated, after the m_free function has been called, if present. Setting m_size to -1 means that the module does not support sub-interpreters, because it has a global state.
• PyMethodDef* m_methods - A pointer to a table of module-level functions, described by PyMethodDef values. Can be NULL if no functions are present. It'll be explained soon.

Now we can define the methods in our module.

static PyMethodDef copy_sign_methods[] = {
    {"copyign", copy_sign_method, METH_VARARGS, "Returns value with the magnitude of x and the sign of y."},
    {NULL, NULL, 0, NULL},
};

In this PyMethodDef structure we define the methods in our module. It is a structure with 4 members.

• The name of the module is the first member. We defined it as copy_sign
• copy_sign_method is the C function to be invoked.
• METH_VARARGS is the third member. This is a flag telling the interpreter the calling convention to be used for the C function.
• self — The module object.
• args — It contains the arguments for the function.
• The last string is the docstring for the function, which explains what the function does.

We can create our function copy_sign_method to perform the copy_sign function now.

static PyObject*
copy_sign_method(PyObject* self, PyObject* args) {
    double x, y, ret;
    if (!PyArg_ParseTuple(args, "dd", &x, &y)) {
        return NULL;
    }
    ret = copysign(x, y);
    return PyFloat_FromDouble(ret);
}

PyObject — The CPython documentation defines PyObject as

This is a type that contains the information Python needs to treat a pointer to an object as an object. A normal “release” build contains only the object’s reference count and a pointer to the corresponding type object. Nothing is actually declared to be a PyObject, but every pointer to a Python object can be cast to a PyObject*.

In short, Every Python Object type is an extension of PyObject type. We can convert any python object to a PyObject.

• First, we define two variables to store the arguments of the function, x, and y. ret is the variable to store the output of the function.
• PyArg_ParseTuple() is the function to parse the arguments from PyObject* args.
• args is the PyObject type that holds the arguments
• "dd", &x, &y implies that we are accepting two double variables and they should be stored in x and y respectively.
• Then the returned value from the function is stored in ret.
• PyFloat_FromDouble() converts double variable in C to a float variable in Python and the value is returned.

Yayy!!! We’ve completed the C part. Now we can package our module and test it.

Building and Testing the Module

Let’s create a setup.py file to build and install the package.

from setuptools import setup, Extension
setup(name="copy_sign",
  version="0.0.1",
  description="Python module for the copysign C function",
  author="author",
  author_email="author@email.com",
  ext_modules=[Extension("copy_sign", sources=["./copy_sign.c"])]
)

• setup.py describes the information about our module.
• sources=["./copy_sign.c"]) is a list of paths to files with the source code.

Let’s build the module.

python setup.py build

and install it…

python setup.py install

It’s time to test our module. Let’s write a simple test for our module in test.py

import unittest
import copy_sign
class Copy_signTests(unittest.TestCase):
    def testCopy_sign(self):
            self.assertEqual(copy_sign.copysign(1, 42), 1.0)
            self.assertEqual(copy_sign.copysign(0., 42), 0.0)
            self.assertEqual(copy_sign.copysign(1., -42), -1.0)
            self.assertEqual(copy_sign.copysign(3, 0.), 3.0)
            self.assertEqual(copy_sign.copysign(4., -0.), -4.0)
            self.assertEqual(copy_sign.copysign(1., 0.), 1.)
if __name__ == "__main__":
    unittest.main()

Run the tests…

python -m unittest test.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK 

Packaging the module

Now let’s package our module.

python setup.py bdist_wheel sdist

This command will create the packaged module in the dist folder inside the main project directory.

Conclusion

That’s It. We’ve created a CPython module, tested and packaged it.