System Overview

This document describes the OpenPET electronics system. The purpose of the OpenPET electronics is to provide a system that can be used by a large variety of users, primarily people who are developing prototype nuclear medical imaging systems. These electronics must be extremely flexible, as the type of detector, camera geometry, definition of event words, and algorithm for creating the event word given the detector outputs will vary from camera to camera. This implies that users must be able to modify the electronics easily, which further implies that they have easy access to documentation, including the schematics and documents needed to fabricate the circuit boards (Gerber files, bill of materials, etc.) and source code (for both firmware and software). They also need support in the form of instructions, user manuals, and a knowledge base, and they want fabricated circuit boards to be readily available.

Thus, the OpenPET electronics system includes hardware, firmware, and software. It is scalable enough to provide solutions ranging from a “test bench” for a small number of detector modules to a complete camera. It is also “open source” to both maximize flexibility and minimize redundant development.

_images/Slide2.PNG

Fig. 1 Schematic of the OpenPET system architecture.

The basic system architecture is shown in Fig. 1. There are four types of custom electronics boards in the system: the Detector Board (DB), the Support Board (SB), the Coincidence Interface Board (CI), and the Multiplexer Board (MB). The Support Board plays two roles in the system, depending on the firmware.

The general data flow is that analog signals from detector modules provide the inputs to the Detector Board. This board processes the analog signals to create a singles event word, which is a digital representation of this single gamma ray interaction. The singles event words are passed to the Support Board loaded with detection firmware, whose main function here is to multiplex the singles event words from multiple Detector Boards. The singles event words are then passed through the Coincidence Interface Board to the Multiplexer Board, which can provide a further layer of multiplexing for singles event words, if necessary. Next the multiplexed singles event words are passed to another Support Board loaded with coincidence firmware, which searches through the singles event words for pairs that are in time coincidence and then forms coincidence event words. These coincidence event words are then passed to the Host PC. Optionally, the Support Board with coincidence firmware can act as a multiplexer and pass unaltered singles event words to the Host PC.

Definitions

The OpenPET components are housed in an assembly whose form factor is the same as a 12-slot VME crate that accommodates 6U boards. A Support Board essentially replaces the backplane of the VME crate and all the other boards plug into it. The plug-in boards have the same form factor as a VME 6U board, except that the position of the connectors is offset (compared to true VME boards) to prevent OpenPET boards from being plugged into standard VME systems and vice versa.

Support Crate

A Support Crate (Fig. 2) is conceptually similar to a VME crate (with controller), namely an intelligent support structure that “functional” boards can be plugged into. It consists of a mechanical frame with 12 plug-in slots, a Support Board (that has a considerable amount of programmable processing power and also acts as a backplane), power supplies, cooling fans, and appropriate boards plugged into slots 9-11. Slots 0-8 are vacant. Slot 9 holds a Host PC Interface Board, which is used to communicate with the Host PC; this board is optional. Slot 10 holds a User IO Board, which allows users to interface to external components such as EKG signals and motor controllers; this board is optional. Slot 11 holds a Debugging Board, which has interfaces to logic analyzers, a number of diagnostic LEDs, an external clock input, and a JTAG connector; this board is optional. Some ancillary components (such as DRAM memory and a QuickUSB board) are also necessary for a functioning Support Crate. By programming the Support Board with appropriate (but different) firmware, the Support Crate becomes part of either a Detector Unit or a Coincidence Unit.

Detector Unit

A Detector Unit (DU), as shown in Fig. 2 consists of a Support Crate with between one and eight Detector Boards plugged into slots 0-7. Each Detector Board can process up to 32 analog input signals. A Detector Unit can therefore process up to 256 analog signals, which corresponds to 64 conventional block detector modules (with 4 analog outputs per module). In a Small System (Fig. 4), Slot 8 is empty (if data is transferred to the Host PC through USB or Ethernet via the Host PC Interface Board plugged into Slot 9). In a Standard (Fig. 5) or Large System (Fig. 6), a Coincidence Interface Board must be plugged into Slot 8 of the Detector Unit. There are two versions of the Coincidence Interface Board: Coincidence Interface Board-1 (CI-1) for the Standard System and Coincidence Interface Board-8 (CI-8) for the Large System. At present, the Coincidence Interface Board-8 has not been designed or specified. These boards transfer event data and bidirectional control data between the Detector Unit and a Coincidence Unit.

_images/Slide78.PNG

Fig. 2 Support Crate (left) and Detector Unit (right). A Detector Unit is a Support Crate with up to 8 Detector Boards (in Slots 0-7). For a Small System, Slot 8 is usually empty, although a Coincidence Interface Board can (optionally) be plugged into it. For Standard and Large Systems, a Coincidence Interface Board must be plugged into Slot 8.

In a Small System (see Small System), the Support Board in the Detector Unit is programmed to multiplex outputs from the Detector Boards, process coincident events, and pass the coincident events to the Host PC. It can also be programmed to multiplex singles events and pass them to the Host PC. In a Standard or Large System (see Standard System and Large System), the Support Board in the Detector Unit is programmed to multiplex singles events from the Detector Boards and forward them to a Coincidence Unit.

Coincidence Unit

In Small Systems, the coincidence processing is performed on the Detector Unit’s Support Board. In Standard and Large Systems, the coincidence processing is performed in a Coincidence Unit (CU) as shown in Fig. 3.

The Coincidence Unit for a Standard System consists of a Support Crate with between one and eight Multiplexer Boards plugged into slots 0-7. The Support Board is loaded with firmware to perform the coincidence processing. Each Multiplexer Board communicates with one Detector Unit via the Coincidence Interface Board using a cable. Similar to the Coincidence Interface Board, there are two versions of the Multiplexer Board: Multiplexer Board-1 (MB-1) for the Standard System and Multiplexer Board-8 (MB-8) for the Large System. At present, the Multiplexer Board-8 has not been designed or specified. The Coincidence Unit’s Support Board is programmed to do the coincidence processing and pass the coincident events to the Host PC, although it can also function as a multiplexer and forward singles events. Data is transferred to the Host PC either through USB or Ethernet via the Host PC Interface Board plugged into Slot 9.

In a Coincidence Unit, the Support Board that acts as a backplane for the Support Crate is loaded with coincidence firmware. In this case, the Support Board with related firmware and software is called a Coincidence Unit Controller (CUC).

In the Coincidence Unit for the Standard System, each MB-1 connects with only one Detector Unit via a single cable, allowing up to 64 Detector Boards (or 512 block detector modules) in the system. In the Coincidence Unit for a Large System, the MB-8s plugged into slots 0-7 connect via cables (one cable per Detector Unit) with up to 8 Detector Units, allowing up to 512 Detector Boards (or 4096 block detector modules) in the system. The MB-8s are programmed to serve as multiplexers for events coming from up to 8 Detector Units. Due to the nature of multiplexing, this allows a larger number of channels to be serviced, but does not increase the maximum total event rate (singles or coincidence).

_images/Slide79.PNG

Fig. 3 Coincidence Unit for a Standard System (left) and a Large System (right). For both Standard and Large Systems, a Coincidence Unit is a Support Crate with up to 8 Multiplexer Boards (in slots 0-7). In a Large System, the Multiplexer Boards function as multiplexers.

System Configuration

OpenPET can be configured either as a Small System, Standard System, or Large System, with the difference primarily due to the number of analog signals that can be read out. To determine which system components you need, the first step is to determine how many DBs are necessary. Each DB can process up to 16 or 32 analog signals (depending on the DB used, mixing of different types of DB is not supported), so the minimum number of DBs necessary is the number of analog signals divided by 16 or 32. While information can be shared between DBs, processing is far easier if all the signals from the same detector module are on the same DB. Thus, if your detector module has 5 analog outputs and you use a 32-channel DB, it is easiest to have each DB process 30 analog signals (i.e., from six detector modules) and not use the other two channels on each Detector Board. Thus, the initial estimate for the number of DBs needed is the number of analog signals divided by the number of analog signals you will have each DB process. This estimate may be modified due to the camera topology, as described in the following subsections. Once you have determined this initial estimate for the number of Detector Boards, you can determine whether you will need a Small, Standard, or Large OpenPET System.

Small System

If the total number of Detector Boards is 8 or fewer, you can use a Small System (Fig. 4). This consists of a single Detector Unit (defined in Detector Unit) connected to a Host PC. A Detector Unit (DU) can have anywhere between 1 and 8 DBs plugged into it. The initial estimate of the number of DBs in your system may need to be increased to support the camera topology. In the Small System, the default coincidence processing algorithm searches for coincidences between singles events that originate on different DBs, but it doesn’t allow coincidences between singles events that originate on the same DB. Thus, more DBs may be necessary to allow all the desired coincidences, as none of the modules in a DB can be in coincidence with each other.

As an example, consider a four-headed PET system, where each head consists of a 3x3 array of detector modules and each detector module has five analog outputs. The OpenPET system would be configured as follows. Each 32-channel DB would service 6 detector modules, using 30 analog channels and leaving two channels on each DB unused. As the system consists of 36 detector modules (four heads of nine modules each), the initial estimate for the number of DBs is six. To see whether the appropriate coincidences can be accommodated, we first try to distribute the detector modules as follows. DB 0 services six modules from head 0, DB 1 services three modules from head 0 and three from head 1, DB 2 services six modules from head 1, DB 3 services six modules from head 2, DB 4 services three modules from head 2 and three from head 3, and DB 5 services six modules from head 3. Unfortunately, this will not work, as two Detector Boards (numbers 1 and 4) service modules from two different heads, which means that the system will not look for all possible coincidences between detector modules that are in different heads. No amount of redistributing the modules among the DBs will satisfy these criteria either. Thus, the only way the coincidence criteria can be satisfied (using the default coincidence processing software) is to use two DBs per head, for a total of eight Detector Boards. While the coincidence processing software can be rewritten to allow coincidences between two singles events that originate in the same DB, this is likely to take significantly longer and cost more than purchasing two additional Detector Boards.

_images/Slide16.PNG

Fig. 4 Configuration of a Small OpenPET System.

Standard System

If the total number of DBs is between 9 and 64, you can use a Standard System (Fig. 5). This consists of between 2 and 8 DUs (defined in Detector Unit) and a single CU (defined in Coincidence Unit) connected to a Host PC. The DUs are connected to the CU via the Coincidence Interface Board CI-1 in each of the DUs and the Multiplexer Board MB-1 in the CU. Determining the number of DUs and DBs per DU needed follows the same principles as described in the Small System section. Each DU should only contain modules that will not be in coincidence with each other, as the default coincidence processing software does not allow coincidences between detectors that originate from the same DU. The DU should then contain the minimum number of DBs necessary to service all the required detector modules.

Each DU services a maximum of eight DBs, and each 32-channel DB services a maximum of 32 analog inputs (note that a conventional PET block detector has four analog outputs, one for each photomultiplier tube). Thus, the Standard System can support a maximum of 2048 analog inputs (32 analog channels per DB, 8 DBs per DU, and 8 DUs per CU), which corresponds to 512 block detector modules.

As an example, consider a cylindrical PET camera, where each detector module has five analog outputs and covers a 5 cm x 5 cm area. There are 44 detector modules per ring (roughly 70 cm diameter) and 5 rings (25 cm axial coverage). The OpenPET system would be configured as follows. The system consists of 220 detector modules. Each 32-channel DB would service 6 detector modules, using 30 analog channels and the remaining two channels on each DB would be unused. As there are a maximum of eight DBs per DU, a DU can service a maximum of 48 of these detector modules. If we divide the 220 modules by 48 modules per DU, we find that the system requires 4.583 DUs. Since DUs are quantized, it really needs 5 DUs, with each DU servicing 44 detector modules. In order to make sure that the correct coincident pairs will be collected, the modules in each DU should be selected so that each DU services a “pie slice” that spans ~72° azimuthally and the full 25 cm axial thickness.

_images/Slide17.PNG

Fig. 5 Configuration of a Standard OpenPET System. Each of slots 0-7 in the Coincidence Unit contains a Multiplexer Board-1 that services a single DU. Slot 8 in the Detector Unit contains a Coincidence Interface Board-1 that transfers event data and bidirectional control between the DU and CU.

Large System

If the total number of DBs is between 65 and 512, you must use a Large System (Fig. 6). This consists of between 9 and 64 DUs (defined in Detector Unit) and a single CU (defined in Coincidence Unit) connected to a Host PC. The DUs are connected to the CU via the CI-8 in each of the DUs and the MB-8 in the CU. Determining the number of DUs and DBs per DU needed follows the same principles as described in the Standard System section. The difference between the Standard and Large Systems is that the Multiplexer Board-8 that plugs into slots 0-7 of the CU contains active circuitries (i.e., FPGA, etc.) that are configured as multiplexers. This allows each of the eight slots in the CU to service up to eight DUs (in a Standard System, each CU slot services a single DU). Thus, the Large System can support a maximum of 16,384 analog inputs (32 analog channels per DB, 8 DBs per DU, and 64 DUs per CU), which corresponds to 4,096 block detector modules. Again, the group of DUs processed by one slot in the CU should only contain modules that will not be in coincidence with each other, as the default coincidence processing software does not allow coincidences between detectors serviced by the same CU slot. The DU should then contain the minimum number of DBs necessary to service all the required detector modules.

_images/Slide18.PNG

Fig. 6 Configuration of a Large OpenPET System. Each of slots 0-7 in the CU contains a Multiplexer Board-8 that operates as a multiplexer, allowing that slot to service between 1 and 8 DUs. Slot 8 in the Detector Unit contains a Coincidence Interface Board-8 that transfers event data and bidirectional control between the DU and CU.

Timing & Timing Signals

The system level timing signals are shown in Fig. 7. There are two timing signals–the system clock, which is an 80 MHz clock signal, and the time slice boundary, which defines the beginning of a time slice. The firmware will support both “short” and “long” event words. In “short” mode, the time slice boundary is generated every eight system clock cycles, while in “long” mode it is generated every sixteen system clock cycles. The choice creates a tradeoff–in “short” mode the dead time is a factor of two shorter, but the number of bits per event word is also a factor of two smaller.

The general concept is that the system divides time into small, fixed length time slices (100-200 ns or 8-16 clocks). All individual operations must occur within a single time slice, which implies that only singles event words that occur in the same time slice can be combined to form a coincident event. Since it can take significantly longer than a single time slice to fully process a single event, the system is pipelined so that the processing is divided into smaller steps that each can be completed in a single time slice.

During one time slice, each of the boards in a Standard System that output singles event words (i.e., DB, CI-1, and MB-1) can pass four singles event words. Thus, the maximum singles rate seen at the CI-1 output of each Detector Unit is 4 singles event words per time slice, or approximately 40 million “short” singles event words per second. Thus, 32 singles event words (four for each of the eight CI-1) enter the Coincidence Unit per time slice, or approximately 320 million “short” singles event words per second. In a Standard System, there are 8 Detector Units with 28 possible Detector Unit - Detector Unit combinations. So theoretically the Coincidence Unit can identify 448 coincident events per time slice (16 for each of the 28 Detector Unit-Detector Unit combinations), corresponding to 4.48 billion coincidence event words per second. In practice, the maximum event rate is limited by the transfer rate between the Coincidence Unit and the Host PC, which is considerably slower.

System Clock

The system clock is an 80 MHz clock. In general, it is generated on the Support Board in the Coincidence Unit (although it can be generated on the Support Board in a Detector Unit such as in the Small System), and then buffered through the rest of the system. Propagation delays will introduce skewing, therefore each FPGA that outputs data will also output a copy of the system clock that is synchronized with its output data signals. In general, each board in the system regenerates the clock using a phase-locked loop (PLL) in order to maintain signal quality and to minimize phase drift.

_images/Slide5.PNG

Fig. 7 System level timing signals.

Time Slice Boundary

The rising edge of the time slice boundary defines the beginning of a time slice. The width of the pulse is one system clock cycle, and the period is eight system clock cycles (for “short” event word mode) or sixteen system clock cycles (for “long” event word mode). In general, it is generated on the Support Board in the Coincidence Unit (although it can be generated on the Support Board in a Detector Unit such as in the Small System), and then buffered through the rest of the system. Propagation delays will introduce skewing; therefore each FPGA that outputs data will also output a copy of the time slice boundary that is synchronized with its output data signals.

Time Slice

The system divides time into small, fixed length time slices (100-200 ns or 8-16 clocks). All individual data processing operations must occur within a single time slice, which implies that only singles event words that occur in the same time slice can be combined to form a coincident event. While it can take significantly longer than one time slice to fully process a single event, the system is pipelined so that the processing is divided into smaller operations that each can be completed in a single time slice. It takes one time slice to transfer a singles event word.

Firmware & Software Structures

{Write this after you’ve written the corresponding section for the Developers Guide?}

The OpenPET firmware and software structures are based on a computer network tree topology. The configuration strategy needs to fulfill the following two basic requirements:

(1) Compatibility with different types of detector modules (e.g. single analog channel addressing, single crystal addressing for a conventional block detector, etc.);

(2) Compatibility with different sized systems (e.g., Small, Standard and Large Systems).

In addition, the addressing strategy needs to be implementable, flexible and reliable.

Standard System

_images/FirmwareSoftwareStructure_Standard.png

Fig. 8 Standard system firmware and software structure. (Add Host PC Interface Board, like Fig. 6 Developer Guide.)

Large System

Small System

As described earlier (Small System), the OpenPET system can also be configured as a Small System. In a Small System, a support board is configured as a Coincidence Detector Unit Controller (CDUC), which interfaces with the detector boards and performs coincidence functions. Basically the CDUC performs the functions of both the CUC and DUC. The initial firmware and software for the first release has been developed for a Small System. The configuration for a Small System is shown in Figure ?.