Pulsar post processing tools 3pt.exe and 3pt-plot.py
This software tool can be used to extract pulsar signals from 8bit unsigned IQ data captured with a RTL dongle or equivalent.
First the data has to be captured with the tool rtl_sdr.exe as decribed in http://parac.eu/projectmk17.htm
example; rtl_sdr -s 2e6 -f 419.0e6 -n 2e8 dump.bin, meaning;
-s 2e6; sample rate 2MS/s
-f 419.0e6; frequency 419MHz
-n 2e8; number of samples to process
dump.bin; name of the file
Next the pulsar has to be extracted from that dump.bin and that can be tricky.
From the many tools available we have chosen to write our own.
Description of the method
Short description: 3pt.exe chops the I&Q file into pulsar periods performs FFT and writes them as spectrum files to disk.
The python script reads the spectrum files and folds them to one file and plots it in a multi plot graph.
The 3pt tool takes the parameters from a configuration file 3pt.cnf.
Here you can set all the needed parameters and next run 3pt.exe.
The first data strings are taken from the I&Q file are used to calculate the pass band of the radio.
This passband is used as a correction to make the passband flat in frequency response.
This passband file is also written to disk (FYI). Also you will notice the 3pt.log file.
Next the scanning of the I&Q file starts again and a chunk of samples is taken to fill a bin and the spectrum amplitudes are corrected with a bandpass correction.
Next the baseband corrected bin spectrum is split up into 10 channels and the dispersion correction is applied.
The 11th channel is used to add all the 10 same bins together in a sum file.
Next the 10 channel values are individually appended to the next bin. When all the bins in one period are determined, then one period file with 11 channels is written to disk.
This is done for all the periods.
You can read these files in Excel or Libre office, but you also can use the python script 3pt-plot.py.
-Put the 3pt.exe, the IQ file the 3pt.cnf and the python plot script in one directory.
-Open the config file, edit it, and save it.
-Run 3pt.exe. One processor core of your PC will be occupied completely.
-When finished a number of period files will be written:
1 spectrum file containing 10 spectrum rows and 1 sum row each.
1 band pass file; FYI.
1 band pass correction file; FYI.
-The files can be examened with a spread sheet program or with the python plot script.
-You could want to plot individual period files; this can be usefull when you have a strong signal.
Also this is usefull to notice time jumps in your signal caused by PC interrupts.
Further, you can now notice individual periods drowned in RFI. You can exclude these useless periods from your total average.
Clipping RFI filtering option.
-Clipping can be activated in line 12 and a clipping level can be set in line 13.
0: no clipping.
1: use Hanning window when performing FFT.
2: use clipping in the form of y=x^B. The value "B" is given in row 13.
When "B"=0.5, then the output y is a sqrt(x). "B" may be set from >0 to 1.
3: use Hanning and clipping together
Optimal period/sample rate searching.
There are parameters wich influence the folding of the pulsar signals. These are period time, sample rate and DM.
The priod time can be calculated very precise using for instance TEMPO or online:
The DM and the observing radio frequency must be known so a correct de-dispersion can be performed.
The sample rate should be known, but often is not accurately known. Also temperature drift shifts the sample rate. Best is to use a GPS diciplined oscillator.
So, if the standard x-tal in the dongle is used, the sample rate set is not exactly the sample rate executed.
To find the correct sample rate, a search for the best result can be done.
In the config file set the search value (in row 4) from 0 (=no search) to 0.005 (=0.005 pro mil) of the set sample rate to vary.
For instance when the set sample rate is 2000000Sps, then with 0.005 the application will vary the sample rate from
2000000+0.005*2000000=1999980 to 2000010 Sps in 10 steps and record the results. The result files are written to disk and can be read with notepad or the same python script.
The calculation can take a long time..upto many hours or days, so set the app search not over 200 process periods max.(line 11)
Choose a low bin value; smaller then 50, because else the routine that tries to find the maximum peak, can mistakenly select a RFI peak.
The config file
0.------first part calculation------------
1.pulsar name = B0329+54
2.pulsar period time [s]= 0.71448358541
3.Sample rate [Sps]= 2000000.0
4.search optimal sample rate [ppm] (default step:0.005, 0:off)=0
6.center freq [MHz]= 404.0
7.IQ input file name =da3.bin
8.number of output bins= 108
9.output file name without extension= spec
10.process period from= 0
11.process period to (0:all)= 1118
12.clipping & RFI filtering (0:no,1:yes)= 0
13.clipping & RFI filtering level [%]=1
14--------second part folding, plotting----------
15.plot period from = 0
16.plot period to (0:all)= 95
17.fold period from =2
18.fold period to (0:all)=1118
-1. The period time of the pulsar can be calculated with TEMPO like described here
Or you can use the on line calculator here
-2. The Dispersion Measure or DM can be found here
-3. If in the config file the "to" period is set to 0, then all the periods will be processed.
This is usefull when the number of period is not yet known.
-4. Any number of bins can be set but advised is a number between 10 and 1000. For B0329+54 optimal is 108 bins.
For optimal bin size see also http://neutronstar.joataman.net/technical/epoch_folding.html
-5. By writing the period files to disk (like with CFRAD2), the advantage is that you can quickly select another range of periods.
In that way you can scan the entire set to check if the pulsar is strong in one section and weak in another section of the captured file.
This can occur when you have done a passive scan, where the pulsar sweeps through the beam of the antenna.
The disadvantage is that the number of files kan be huge.
-6. The data IQ files can be huge, so to start a post processing session you can begin with processing only the first 10 periods and writing them to disk.
Now you can examen the 10 files and decide how to poceed.
-7. If you have a realy big IQ file to process, you can decide to use all your PC processor cores. You can rename the 3pt.exe to 3pt1.exe, 3pt2.exe etc.
Do likewise for the 3pt.config files, and edit them to process a chunk of your big file; like 3pt1; process from 0 to 1000, next 3pt2 process 1000 to 2000, etc.
Then start all the executables.
-8. It appeares that when using the RTL dongle, the data of the first two periods is misformed. It is adviced to skip those and not add them and use them for pulse formation.
You can do that by setting 'fold period from' to 2 or 3 in the config file.
-9. The 10 spectra channels are usefull for checking if the pulsar is indeed present.
Because of the de-dispersion correction, the channels are shifted in time. When looking at a bump in one of the channels, the same bump must be in line with all the channels for a confirmation of the pulsar.
If the bumps are shifted (and the correct DM has been set) then the bump is not a pulsar.
-10. When starting a session, be sure that the exact sample rate is known and stable. Remember that sample rate and pulsar period are connected. For a chosen sample rate, the product of Sps*period time=constant.
You also can find that constant in the logfile. So when you are confident that the calculated period time is correct but you are not confident of the samplerate, then change the SPS and find the pulsar peak value.
Now you can use the calibrated sample rate also for other pulsar measurements.
Also; it is not usefull to perform a very long measuring session when the sample rate is not stable. The pulsar peaks will be smeared and folding them give no enhancement.
-11. When doing pulsar analyses, it is usefull to also plot the frequency channels, like with this tool, so you can judge the quality of them.
When one or more channels displays severe RFI you can decide not to use that channel and so improve the S/N ratio.
This is also automatically done by the script. Two signal to noise values are given. The first one is calculated when all the periods are folded.
The second one takes the peak value of the first run and folds only the periods together which give an improvement of the s/n ratio.
To further judge the capturing session, the efficiency factor of the periods with s/n ratio>1 is given.
-12. When experimenting with other settings, it is adviced to alter the output spectrum file name, because else they will be overwritten.
-13. The pulsar B0329+54 has two different mode states; normal or abnormal. They can be calculated also. see att paper.
The answer for the example file, or the complete dump2000a.bin file, should be R=0.4; this is the normal mode.
Work in progress.
0437-47 IQ data and info
vela1 IQ data and info
Michiel Klaassen july 2020