This is a draft of the April-June 2014 status report. Please check back after it is finalized, and an announcement email is sent to the FreeBSD-Announce mailing list.
This report covers &os;-related projects between April and June 2014. This is the second of four reports planned for 2014.
The first quarter of 2014 was, again, a hectic and productive time for &os;. The Ports team released their landmark first quarterlystablebranch. &os; continues to grow on the ARM architecture, now running on an ARM-based Chromebook. SMP is now possible on multi-core ARM systems. bhyve, the native &os; hypervisor, continues to improve. An integral test suite is taking shape, and the Jenkins Continuous Integration system has been implemented. &os; patches to GCC are being
forward-ported, and LLDB, the Clang/LLVM debugger is being ported. Desktop use has also seen improvements, with work on Gnome, KDE, Xfce, KMS video drivers, X.org, and vt, the new console driver which supports KMS and Unicode. Linux and Wine binary compatibility layers have been improved. UEFI booting support has been merged to head. The &os; Foundation continues to assist in moving &os; forward, sponsoring conferences and meetings and numerous development projects. And these are only some of the things that happened! Read on for even more. ?>
Thanks to all the reporters for the excellent work! This report contains 20 entries and we hope you enjoy reading it.
The deadline for submissions covering between July and September 2014 is October 7th, 2014.
The so-called "CUSE4BSD" has been imported into the base system of &os;-11. CUSE is short for character device in userspace. The CUSE library is a wrapper for the devfs(8) kernel functionality which is exposed through /dev/cuse. In order to function, the CUSE kernel code must either be enabled in the kernel configuration file or loaded separately as a module. Follow the commit message link to get more information.
The &os; RPC stack, used as a base for its NFS server, received multiple optimizations to improve performance and SMP scalability. Algorithmic optimizations reduced processing overhead, while improved locking allowed it to scale up to at least 40 processor cores without significant lock congestion. Combined with some other kernel optimizations, the peak NFS request rate increased by many times, reaching up to 600K requests per second on modern hardware.
The CAM Target Layer (CTL), used as base for new kernel iSCSI server, also received a series of locking optimizations which allowed its peak request rate to increase from ~200K to ~600K IOPS with the potential of reaching a rate of 1M requests per second. That rate is sufficient to completely saturate 2x10Gbit Ethernet links with 4KB requests. For comparison, the port of net/istgt (user-level iSCSI server) on the same hardware with an equivalent configuration showed only 100K IOPS.
There is also ongoing work on improving CTL functionality. It was already made to support three of four VMware VAAI storage acceleration primitives (net/istgt supports 2), while the goal is to reach full VAAI support during next months.
With all these improvements, and earlier improvements in CAM, GEOM, ZFS, and a number of other kernel areas coming soon, FreeBSD 10.1 may become the fastest storage release ever. ;)
These projects are sponsored by iXsystems, Inc.
Arm64 is the name of the in-progress port of &os; to the ARMv8 CPU when it is in AArch64 mode. Until recently, all ARM CPU designs were 32-bit only. With the introduction of the ARMv8 architecture, ARM has added a new 64-bit mode. This new mode has been named AArch64.
Booting &os; on the ARM Foundation Model has made a lot of progress since the last status report. An initial pmap implementation has been written. With this, &os; is able to enter the Machine Independent boot code. The required autoconf functions have been added allowing &os; to start scheduling tasks. Finally the cpu_switch and copystr functions were added. With these two, &os; will boot to the mountroot prompt.
Work has started on supporting exceptions, including interrupts. This will allow more developers to start working on device drivers.
The vt(4) (aka Newcons) project provides a replacement for the legacy syscons system console. It brings a number of improvements, including better integration with graphics modes and broader character set support.
Since the last report, vt(4) gained the ability to make early driver selection. vt(4) selects the best successfully-probed driver before most other kernel subsystems. Also, to make easy migration from syscons(4) to vt(4), multiple virtual terminal subsystems in the kernel are now supported. It is controlled by a small module with just one kernel environment variable. Users can select the virtual terminal system to use by setting kern.vty=sc or kern.vty=vt.
The GENERIC kernel configuration for the amd64 and i386 platforms now includes both syscons(4) and vt(4) by default. This configuration is also planned to be in the next 10-STABLE release and &os; 10.1-RELEASE.
The project finally received a man page, so now vt(4) is not only the project name, but also a link to its documentation. Great thanks to &a.wblock; for that.
Major highlights:
Brief status of supported architectures and hardware:
The ports-mgmt/poudriere-devel port is aware of how to build ports via an emulator. Configuration of the miscellaneous binary image activator is required prior to a poudriere-devel run.
ARMV6, MIPS32 and MIPS64 packages can be produced via full emulation. There are several packages that block a full run of builds. They can be viewed on the "Status of ports building" link.
On current or latest stable/10:
Clone the
./configure --static \
--target-list="arm-bsd-user i386-bsd-user \
mips-bsd-user mips64-bsd-user mips64el-bsd-user \
mipsel-bsd-user ppc-bsd-user ppc64-bsd-user sparc-bsd-user \
sparc64-bsd-user x86_64-bsd-user"
gmake; gmake install
Then set up the binmiscctl tools to do some evil hackery to redirect execution of armv6 binaries to qemu:
binmiscctl add armv6 --interpreter \
"/usr/local/bin/qemu-arm" --magic \
"\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02 \
\x00\x28\x00" --mask "\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff \
\xff\xff\xff\xff\xfe\xff\xff\xff" --size 20 --set-enabled
Install poudriere-devel from ports. It knows how to set up things.
Build poudriere jail to do all the magic:
poudriere jail -c -j 11armv632 -m svn -a armv6 \
-v head
Now run poudriere against that jail to build all the ports:
poudriere bulk -j 11armv632 -a
Nullfs mount the ports tree into the jail:
mkdir /usr/local/poudriere/jails/11armv632/usr/ports
mount -t nullfs /usr/ports /usr/local/poudriere/jails/11armv632/usr/ports
To chroot into the jail:
mount -t devfs devfs /usr/local/poudriere/jails/11armv632/dev
chroot /usr/local/poudriere/jails/11armv632/
We are pleased to announce the availability of conflict-free Python package support across different Python versions based on the USES=uniquefiles feature recently introduced to the Ports framework. A Python package can be marked as buildable and installable in parallel for different Python versions at the same time on the same host. The package building tools, however, do not support this feature yet and the Python team will work closely with portmgr and pkg developers to enable support on a global ports and package scale.
In May and June a huge clean-up operation took place to remove the last bits and pieces targeting easy_install. In the beginning of July we committed the final changes to remove easy_install support completely from the ports framework. This greatly simplifies the infrastructure and allows us to modernize and maintain it with less effort.
We added Python 3.4, removed Python 3.1 after its end of life, updated the setuptools ports to version 5.1 and PyPy's development version to 2.3.1. The latest Python 2.7.8 and an updated setuptools will hit the tree shortly.
Our upstreaming effort continues to produce good outcomes for simplifying maintenance and reducing complexity.
Looking forward, one of the top priorities is to comply with the USES framework in the foreseeable future and to roll out a consistent maintainer policy for integrating new Python-related ports into the tree.
The Unified Extensible Firmware Interface (UEFI) provides boot- and run-time services for x86 and other computers. For the x86 architecture it replaces the legacy BIOS. This project will adapt the &os; loader and kernel boot process for compatibility with UEFI firmware, found on contemporary servers, desktops, and laptops.
Ed and Nathan completed a number of integration tasks over the past three months. Nathan added a first-stage loader, boot1.efi, to support chain-loading the rest of the system from a UFS filesystem. This allows the UEFI boot process to proceed in a similar fashion as with BIOS boot. Nathan also added UEFI support to the &os; installer and release image creation script.
The EFI framebuffer requires the vt(4) system console — a framebuffer driver is not implemented for the legacy syscons(4) console. Ed added automatic vt(4) selection to the UEFI boot path.
The &os; Core Team constitutes the project's "Board of Directors", responsible for deciding the project's overall goals and direction as well as managing specific areas of the &os; project landscape.
Topics for core this quarter have included some far-reaching policy reviews and some significant changes to the project development methodology.
In May, a new release policy was published and presented at the BSDCan developer conference by John Baldwin. The idea is that each major release branch (for example, 10.X) is guaranteed to be supported for at least five years, but individual point releases on each branch, like 10.0-RELEASE, will be issued at regular intervals and only the latest point release will be supported.
Another significant change did not receive approval. When the change to the Bylaws reforming the core team election process was put to the vote of all &os; developers, it failed to reach a quorum.
June saw the culmination of a long running project to replace the project's bug tracking system. As of June 3, the &os; project has switched to Bugzilla as its bug tracking system. All of the history of GNATS PRs has been preserved, so there is no need to re-open old tickets. Work is still going on to replicate some of the integration tweaks that had been applied to GNATS, but all necessary functionality has been implemented and the project is already seeing the benefits of the new capabilities brought by Bugzilla.
An election to select core members for the next two year term of office took place during this period. We would like to thank retiring members of core for their years of service. The new core team provides continuity with previous core teams: about half are incumbents from the previous team, and several former core team members have returned after a hiatus. Core now includes two members of the &os; Foundation board and one other Foundation staff member, aiding greater coordination at the top level of the project. At the same time the core-secretary role was passed on to a new volunteer.
Other activities included providing consultation on licensing terms for software within the &os; source tree, and oversight of changes to the membership of postmaster and clusteradm.
Three new src commit bits were issued during this quarter, and one was taken into safekeeping.
OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources in a datacenter.
OpenContrail is a network virtualization (SDN) solution comprising network controller, virtual router, and analytics engine, which can be integrated with cloud orchestration systems like OpenStack or CloudStack.
This work's goal is to enable &os; as a fully supported compute host for OpenStack using OpenContrail virtualized networking. The main areas of development are:
Since the last report the following items have been completed, which allow for a working demo of OpenStack compute node on a &os; host using OpenContrail solution for network virtualization:
A demo was presented at the DevSummit during BSDCan2014 in Ottawa. Also, a meetup regarding the subject was organized in Krakow, Poland.
Work on this project is sponsored by Juniper Networks.
The &os; Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the &os; Project and community worldwide. Most of the funding is used to support &os; development projects, conferences and developer summits, purchase equipment to grow and improve the &os; infrastructure, and provide legal support for the Project.
We published our third issue of the &os; Journal. We have over 2700 subscriptions so far. We continued working on the digital edition, which will allow subscribers to read the magazine in different web browsers, including those than run on &os;. This will be available for the July/August issue of the Journal.
We hired Anne Dickison, on a freelance basis, as our new marketing director, to help us promote the Foundation and Project.
The annual board meeting was held in Ottawa, Canada, in May. Directors and officers were elected, and did some long term planning. We worked on our vision, core values, project road mapping, and our near term goals. We also met with the core team to discuss roles and responsibilities, project roadmapping, and what we can do to help the Project more.
We were a Gold+ sponsor for BSDCan, May 16-17 and provided 7 travel grants for developers to attend the conference. We also were the sponsor for both the developer and vendor summits.
Justin Gibbs gave a &os; presentation at a &os; user's internal technology summit. Company visits like this help users understand the Project structure better and gives us a chance to communicate what &os; people are working on as well as learn what different companies are doing with &os;, as well as what they'd like to see supported. We can then help facilitate collaboration between the companies and &os; developers.
We were represented at Great Wide Open, April 2-3 (greatwideopen.org), Texas LinuxFest, June 13-14 (texaslinuxfest.org), and SouthEast LinuxFest, June 20-22 (southeastlinuxfest.org).
Hardware was purchased to support an upgrade at Sentex. A new high capacity 1Gbps switch was deployed to allow for more systems to be added to the test lab. The main file server and development box was upgraded to allow more users in the lab simultaneously.
We purchased hardware, including package builders, and a larger server to allow NYI to be a full replica of all Project systems, comparable to what is in place at Yahoo Inc. and ISC.
We worked with our lawyer to create an NDA between the Foundation and individuals for third party NDAs. This allows developers who need access to proprietary documents, to go through the Foundation, via an NDA for access.
&os; Foundation Systems Administrator and Release Engineer, Glen Barber, continued work on producing regularly-updated &os;/arm snapshots for embedded devices, such as the Raspberry Pi, ZedBoard, and BeagleBone.
In addition to producing weekly development snapshots from the head/ and stable/ branches, with feedback and help from Ed Maste, Glen finished work to produce release images that will, by default, provide debugging files for userland and kernel available on the &os; Project FTP mirrors. Note that the debugging files will not be included on the bootonly.iso, disc1.iso, or dvd1.iso images due to the size of the resulting images.
Foundation staff member Konstantin Belousov completed an investigation into poor performance of PostgreSQL on &os;. This uncovered scalability problems in the &os; kernel, and changes to address these issues are in progress.
Some previously completed Foundation-sponsored projects received enhancements or additional work. The ARM superpages project was completed last year, but is now enabled by default in &os;-CURRENT. Many stability fixes and enhancements have been committed to the in-kernel iSCSI stack. The iSCSI project was released in &os; 10.0. Many stability fixes and enhancements have been committed and will be included in &os; 10.1.
Work continues on the Foundation-sponsored autofs automount daemon, UEFI boot support, the updated vt(4) system video console, virtual machine images, and the Intel graphics driver update. Foundation-sponsored work resulted in 226 commits to &os; over the April to June period.
SDIO is an interface designed as an extension of the existing SD card standard, which allows connecting different peripherals to the host with the standard SD controller. Peripherals currently sold at the general market include WLAN/BT modules, cameras, fingerprint readers, and barcode scanners. Additionally, SDIO is used to connect some peripherals in products like Chromebooks and Wandboard. A prototype of the driver for the Marvell SDIO WLAN/BT (Avastar 88W8787) module is also being developed, using the existing Linux driver as the reference.
SDIO card detection and initialization already work, most needed bus methods are implemented and tested.
The WiFi driver is able to load firmware onto the card and initialize it. A rewrite of the MMC stack as a transport layer for the CAM framework is in progress. This will allow utilization of the well-tested CAM locking model and debug features.
The &os; Release Engineering Team is responsible for setting and publishing release schedules for official project releases of &os;, and announcing code freezes and maintaining the respective branches, among other things.
In early May, the &os; 9.3-RELEASE cycle entered the code slush phase. The &os; 9.3-RELEASE cycle is nearing the final phases, and 9.3-RC3 builds will be starting soon. 9.3-RC3 is planned to be the final release candidate for this release cycle, and at the time of this writing, 9.3-RELEASE should be available on schedule.
Work is ongoing to integrate support for embedded architectures into the release build process. At this time, support exists for a number of ARM kernels, in particular the Raspberry Pi, BeagleBone, and WandBoard.
Additionally, work is in progress to produce virtual machine images as part of the release cycle, supporting various cloud services such as Microsoft Azure, Amazon EC2, and Google Compute Engine.
Fiasco.OC belongs to the L4 microkernel family. A microkernel provides a bare minimum of services to the applications running on top of it, unlike traditional kernels that incorporate complex code like IP stacks and device drivers. This allows a dramatic decrease in the amount of code running in the privileged mode of the CPU, achieving higher security while still providing an acceptable level of performance.
Running an operating system kernel on top of the microkernel allows leveraging any software that was developed for that operating system. The OS kernel runs in user-mode side-by-side with other microkernel applications such as real-time components. Multiple OSes, each with their userland applications, can even be run in parallel, thus allowing construction of products where processing of corporate data is strictly separated from the processing of private data.
The project aims to create a port of &os; to the Fiasco.OC microkernel, a high performance L4 microkernel developed by TU Dresden. Existing ports of OpenBSD and Linux are used as a reference. This will allow the use of unique &os; features like ZFS in L4-based projects.
pkg(8) is the new package management tool for &os;. It is now the only supported package management tool for &os; releases from 10.0-RELEASE, including the upcoming 9.3-RELEASE. pkg(8) is available on all currently supported releases. Support for the legacy pkg_tools is due to be discontinued at the beginning of September 2014.
The release of pkg(8) 1.3 is imminent. This includes major improvements in the dependency solver. Now we can:
Beyond the next release, we have work in progress on allowing ranges of versions in dependency rules and handling a selection of "foreign" package repositories, such as CPAN or CTAN or PyPi.
There are plans to use pkg(8) to package up the base system. Along with other benefits, this will allow writing a universal installer: download one installer image and from there install any available version of &os;, including snapshots.
We are also intending to use pkg(8) within the ports tree at package-build time to handle fulfilling build dependencies. This opens the possibility of installing build-dependencies by downloading binary packages, which means you can install a package with customized options with the minimum amount of time spent compiling anything else.
We were generally short on time this quarter. We made less progress than expected on all fronts.
The alternate pkg(8) repository, built with WITH_NEW_XORG, is now available. This alleviates the need for users to rebuild their ports with WITH_NEW_XORG. See the announcement, linked above for further information.
Thanks to a contribution from Jan Kokemüller, Radeon 32bit ioctls are now working on 64bit hosts. This was tested successfully with Wine and StarCraft II on &os; 9.x and 11. This required modifications to emulators/i386-wine-devel so that it works with WITH_NEW_XORG, and the creation of a new port, libtxc_dxtn, to support texture compression required by StarCraft II. We haven't yet had the time to polish everything, so this still requires manual steps.
The DRM generic code update is ready, but it breaks the current i915 driver. Therefore, the i915 driver must be updated before anything is committed.
Compared to the previous status report, OpenCL test programs are running fine now, thanks to upgrades and fixes to libc++ and Clang. Relevant ports are still not ready to hit the ports tree, unfortunately.
Building on the new in-kernel iSCSI target and initiator stack released in FreeBSD 10.0, Chelsio Communications has begun developing an offload interface to take advantage of the hardware offload capabilities of Chelsio T4 and T5 10 and 40 gigabit Ethernet adapters.
The code currently implements a working prototype of offload for the initiator side, and target side offload should begin shortly. The code will be released under the BSD license and is expected to be completed later in the year and be committed to FreeBSD-HEAD, and will likely ship in a FreeBSD release in early 2015.
Extensive testing of tmpfs(5) using the stress2 kernel test suite was done. The issues found were debugged and fixed.
Most of the problems are related to the bugs in the interaction of vnode and node lifetime, culminating in e.g. unmount races and dotdot lookup bugs.
This project is sponsored by the FreeBSD Foundation.
The analysis of the performance of the latest 9.3 version of the PostgreSQL on &os;-CURRENT was performed. The issues which prevented the good scalability on a 40-core machine were determined, and changes prototyped which solve the bottlenecks.
The URL above provides a paper which contains a detailed explanation of the issues and solutions, together with the graph demonstrating the effects on scalability.
ZFSguru is a multifunctional server appliance with a strong emphasis on storage. ZFSguru began as simple web-interface frontend to ZFS, but has since grown into a &os; derivative with its own infrastructure. The scope of the project has also grown with the inclusion of add-on packages that add functionality beyond the traditional NAS functionality found in similar product like FreeNAS and NAS4Free. ZFSguru aims to be a true multifunctional server appliance that is extremely easy to setup and can unite both novice and more experienced users in a single user interface. The modular nature of the project combats the danger of bloat, whilst still allowing extended functionality to be easily deployed.
Where development in the first quarter of this year brought drag-and-drop permissions for Samba and NFS, development in the second quarter focused on strengthening the infrastructure of the project. A new library and toolkit solution dubbed 'Mesa' is in the works, providing a cleaner foundation to the project. A new master server providing secure remote services is being setup, to be located in a high-speed datacenter. But most importantly, a new system build infrastructure has shown great progress and will soon be able to provide automated system builds to our users. This not only improves the frequency of system releases but also frees much developer time to be spent on different areas of the project.
Furthermore, a new website and forum is being worked at, replacing the old-fashioned website that offers only limited functionality. The new website will be linked to the server database, providing real-time updates about the project.
In addition, a new platform for collaborated development is in the works. A service addon has been created for the GitLab project, which is a drop-in replacement of the popular GitHub website. The choice was made to host our own solution and not rely on GitHub itself. In retrospect this appears to be a good decision. The recent development where GitHub removed projects after DCMA-takedowns being sent is incompatible with the philosophy of free-flow-of-information, which the ZFSguru project is a strong proponent of. By hosting our own solution, we have avoided any dependency on third party projects.
It is expected that after the infrastructure of the project has been revamped, work on the web-interface itself can continue. New functionality such as GuruDB and Service Bulletins provide a tighter connection between the server infrastructure and the web-interface. The Migration Manager is one of the last remaining features still missing in the web-interface. This functionality provides an easy way to upgrade the current system by performing a new clean installation, but migrate all relevant configuration to the new installation. It also allows to 'backup' all system configuration in a single file to be stored on a different machine should things go awry.
A longer version of this status report giving a wider
perspective on the project, can be found
FreeBSD received 39 project proposals this year, many of which were of a very high standard. After a difficult selection process narrowing these down into the slots we had been allocated, a total of 16 projects were selected to participate in Google Summer of Code 2014 with FreeBSD.
Projects selected cover a wide range of areas within FreeBSD, covering both the base system and ports infrastructure, userland and kernel. We have students working on firewall optimisation, ports packaging tools, embedded systems, debugging infrastructure, improved Unicode support, enhancements to the loader and to the installer, and several other areas of work. We are just over half way through the allocated time this year, and are very much looking forward to integrating code produced by these projects into FreeBSD.
This is the tenth time FreeBSD has taken part in Google's Summer of Code, and we are grateful to Google to have accepted us as a participating organisation.
The ports tree slowly approaches the 25,000 ports threshold, while the PR count is slightly below 1800.
In Q2 we added three new committers, took in one commit bit for safe keeping, and reinstated one commit bit.
In May, tabthorpe@ was replaced by culot@ as portmgr secretary, and swills@ became a member of the portmgr team.
Commencing July 1, the third intake portmgr-lurkers@ started active duty on portmgr@ for a four month duration. The next two candidates are wg@ and nivit@.
This quarter also saw the release of the second quarterly branch, namely 2014Q2. This branch was not only built for 10 (as 2014Q1) but for 9 as well (both i386 and amd64).