DSP facts

Autocorrelation
in general:
\[
\phi_{yy}[n, n+m] = \text{E}\{y[n]y[n+m]\}
\]
if stationary
\[
\phi_{yy}[n, n+m] =  \phi_{yy}[m] = \text{E}\{\mathbf{y}_{n+m}\mathbf{y}_n^*\}
\]

Deterministic autocorrelation

\[
c_{hh}[l] = \sum_{k=-\infty}^{\infty}h[k]h[l+k] = h[n]*h[-n]
\]
\[
C_{hh}(e^{j\omega}) = H(e^{j\omega})H^*(e^{j\omega}) = |H(e^{j\omega})|^2
\]

Response of LTI system to random input
\[
\Phi_{yy}(e^{j\omega})=C_{hh}(e^{j\omega})\Phi_{xx}(e^{j\omega})
\]

Mean
$m_{\mathbf{x}_n}} = \text{E}\{\mathbf{x}_n\}$
if stationary $m_{\mathbf{x}_n}}=m_x \quad \text{for all } n$
Mean-squared (average power)
$\text{E}\{\mathbf{x}_n\mathbf{x}_n^*\} = \text{E}\{|\mathbf{x}_n|^2\} $
if stationary $\text{E}\{\mathbf{x}_n\mathbf{x}_n^*\} = \text{E}\{\mathbf{x}[n+m]\mathbf{x}[n]\}|_{m=0} = \phi_{xx}[0]$

Inverse DTFT
\[
x[n]=\frac{1}{2\pi} \int_{-\pi}^{\pi} X(e^{j\omega})e^{j\omega n}d\omega
\]
DTFT
\[
X(e^{j\omega}) = \sum_{n=-\infty}^{\infty}x[n]e^{j\omega n}
\]

Partial Derivative Logistic Regression Cost Function

Logistic regression is used for classification problems.  As Andrew said, it's a bit confusing given the "regression" in the name.

LR cost function is given by:
\[
\text{Cost} (h_\theta (x),y) =
\begin{cases}
 -\log(h_\theta (x)) \quad &\text{if } y=1 \\
 -\log(1-h_\theta (x)) \quad &\text{if } y=0
\end{cases}
\]

where \(h_\theta(x) = \frac{1}{1+e^{-\theta^Tx}}\) is the logistic function.

Since \(y \in \{0,1\}\) only, we can reduce the cost function to an equivalent, single equation.
\[
\text{Cost} (h_\theta (x),y) =  -y\log(h_\theta (x)) - (1-y)\log(1-h_\theta (x))
\]

This leads to the overall cost function for the logistic regression:
\[
J(\theta) = -\frac{1}{m} [\sum_{i=1}^m y^{(i)}\log h_\theta(x^{(i)}) + (1-y^{(i)})\log(1-h_\theta (x))]
\]

Our goal is to find \(\min_\theta J(\theta)\). To do so, we use gradient descent, but we first need to find the partial derivatives \(\frac{\partial}{\partial \theta_j} J(\theta)\).

We're going to make use of a neat property of the logistic function:
\begin{align}
g'(z) &= \frac{d}{dz} \frac{1}{1+e^{-z}} = \frac{1}{(1+e^{-z})^2}e^{-z} \\
 &= \frac{1+e^{-z}-1}{(1+e^{-z})^2} = \frac{1}{1+e^{-z}}-\frac{1}{(1+e^{-z})^2} = \frac{1}{1+e^{-z}}(1-\frac{1}{1+e^{-z}}) \\
 &= g(z) (1-g(z))
\end{align}

So for our with our cost function:
\begin{align}
\frac{\partial}{\partial \theta_j} J(\theta) &= -\frac{1}{m} \left [\frac{\partial}{\partial \theta_j} \sum_{i=1}^m y^{(i)}\log h_\theta(x^{(i)}) + (1-y^{(i)})\log(1-h_\theta (x)) \right] \\
 &= -\frac{1}{m} \left [ \sum_{i=1}^m y^{(i)}\frac{1}{h_\theta(x^{(i)})}\frac{\partial}{\partial \theta_j}h_\theta(x^{(i)}) + (1-y^{(i)})\frac{1}{1-h_\theta(x^{(i)})}\left (-\frac{\partial}{\partial \theta_j}h_\theta (x^{(i)})\right) \right]
\end{align}

using the chain rule and the logistic regression derivative, we see that
\begin{align}
\frac{\partial}{\partial \theta_j} J(\theta) &=-\frac{1}{m} \left [ \sum_{i=1}^m y^{(i)}\frac{x_j^{(i)}}{h_\theta(x^{(i)})}h_\theta(x^{(i)})(1-h_\theta(x^{(i)})) - (1-y^{(i)})\frac{x_j^{(i)}}{1-h_\theta(x^{(i)})}h_\theta (x)(1-h_\theta(x^{(i)})) \right] \\
&=  -\frac{1}{m} \left [ \sum_{i=1}^m y^{(i)}x_j^{(i)}(1-h_\theta(x^{(i)})) - (1-y^{(i)})x_j^{(i)}h_\theta (x^{(i)})) \right] \\
&=  -\frac{1}{m} \left [ \sum_{i=1}^m y^{(i)}x_j^{(i)} - x_jh_\theta (x^{(i)}) \right] \\
\frac{\partial}{\partial \theta_j} J(\theta) &=  \frac{1}{m} \sum_{i=1}^m  (h_\theta (x^{(i)}) -  y^{(i)}) x_j^{(i)}
\end{align}

This formula can now be used in gradient descent.

cleaning up unwanted data files from a git repository

I recently made the mistake of merging unwanted data files into the lab git repository, and unfortunately and these files weren't discovered until after they were also pushed to the remote repository.

To clean up the git repository, I used the git filter-branch command, force pushed the changes to the remote, and then had collaborators rebase their local copies of the contaminated branch.

Using git filter branch:

git filter-branch --index-filter 'git rm --cached --ignore-unmatch [filename]' --[commit to begin with]^..
rm -Rf .git/refs/original
rm -Rf .git/logs
git gc

Force pushing the changes:
git push --force origin [branch name]

Have collaborators rebase their local branches
git fetch (NOT PULL!!!!!)
git rebase --onto origin/[branch] [branch] [local branch ([branch] and anything derived from [branch])]

The last command does a hard reset on of the current branch to origin/[branch].  It then takes the commits from [local branch] and [branch] and then applies them to the current branch.

In addition I learned how to use regex to search for files matching wildcards through subdirectories -- you need to use the escape character \

e.g. to git remove all .txt files from a directory and all subdirectories:
git -rm ./\*.txt
without the escape character, the * wildcard is expanded by the shell.  with the escape character, git rm is allowed to interpret the *.txt

Connecting to wireless via command line

to initialize the interface: sudo ifconfig <interface name like wlan0> up
to connect to wireless: sudo iwconfig <interface name> essid <network name like Stanford>
to obtain an ipaddress: sudo dhclient <interface name>
to test whether connection is valid: ping -c 1 www.google.com

Adding a printer ubuntu

Problem is that nobody remembers the root password, which the system prompts you for upon selecting add printer in either the printers gui or localhost:631

The way around this was to run sudo system-config-printer from the command line...

Synapse Calibration procedure

on ng-thalamu

There are 4 steps to the calibration:

1. gleak
2. Erev
3. txmt
4. tau_lpf

For gleak (Erev)

1. Arrange the board's xml file so the chip and synapse numbers you are looking to calibrate are first in the file
2. Match the chip and synapse numbers in syn_f_vs_g_sleak.py (f_vs_erev_sleak.py) to the desired chip and synapse numbers AND BE SURE TO SAVE.
3. In spring, run  syn_f_vs_g_sleak.py (f_vs_erev_sleak.py)
4. Set the chip and synapse numbers in calib_syn_f_vs_g_sleak.py (calib_f_vs_erev_sleak.py) to the desired chip and synapse numbers AND BE SURE TO SAVE.
5. Run calib_syn_f_vs_g_sleak.py (calib_f_vs_erev_sleak.py) in ipython --pylab
6. Execute extract_syn_param_g_lksoma.py (extract_syn_param_erev_lksoma.py) in ipython --pylab, and then call run_default with the appropriate bif file, board name, chip number, and synapse number.
7. Use median values in histogram as parameters in xml file

For txmt (tau_lpf)

1. In neuro-boa/apps/calibrate_neuron/calibrate_synapse/calibrate_pe/calibrate_pe.cpp (calibrate_vleakpf/calibrate_vleakpf.cpp):
1.  make sure the appropriate chip calibration is pushed back
• CALIBRATION_DAC_FILE.push_back(<check this>);
• CALIBRATION_ADC_FILE.push_back(<check this>);
2. make sure the appropriate chip number is pushed back
• selected_chips.push_back(<chip number>);
3. select the appropriate chip in the for loop around ~line 621
2. run make
3. run calibrate_pe (calibrate_vleakpf)
4. For txmt:
1. change pw_values.csv to pw_values_<board>_<chip>.csv
2. In fit_2d_1coeff.py
• change filename to match data
• select synapse
3. run fit_2d_1coeff.py
4. set C1 to median value
5. set C3 to 0
5. For tau_lpf:
1. change the data/ folder name to data<chip num>/
2. In fit_tau.py
1. change data folder to match chip number
3. run fit_tau.py
4. set tau_lpf to median value

Synaptic Pulse extender

On Neurogrid, the synaptic conductance, \(x\), is governed by
\[\tau \dot{x} = -x + g_{max}\sum_ip(t-t_i)\]
where \(p(t)\) is a square pulse of length \(t_{xmt}\) resulting from spikes arriving at time \(t_i\), \(g_{max}\) is the maximum synaptic conductance, and \(\tau\) is the synaptic time constant.

Let's analyze the steady state conductance induced by a Poisson spike train where the interarrival times of spikes are distributed exponentially with pdf \(f(t) = \lambda e^{-\lambda t}\).

At steady state, \(\dot{x} = 0\), and \(0 = -x + g_{max}p(t)\).
Therefore at steady state, \(x = g_{max}p(t)\) and on average \(\langle x \rangle=g_{max} \langle p(t) \rangle\).

There is one wrinkle in our analysis: when a spike arrives within \(t_{xmt}\) of the previous spike. The two resulting pulses do not add linearly.  The second pulse merely extends the previous pulse by the time between the spikes.

\[\langle p(t)\rangle=\langle \mathrm{spike\ rate}\rangle \langle \mathrm{average\ pulse\ value}\rangle\]

For a Poisson process, the rate is simply \(\lambda\).

For a full pulse, the area is simply \(t_{xmt}\).  For the pulse resulting from a collision, the area is \(t=\Delta t\). So

\[\langle \mathrm{average\ pulse\ value}\rangle = \int_0^{t_{xmt}} t \lambda e^{-\lambda t} dt + \int_{t_{xmt}}^\infty t_{xmt}\lambda e^{-\lambda t} dt\]

\[ = \left. t e^{-\lambda t}\right|_{t_{xmt}}^0 + \int_0^{t_{xmt}} e^{-\lambda t} dt + \left. t_{xmt}\lambda e^{-\lambda t} \right|_\infty^{t_{xmt}}\]
\[ = -t_{xmt} e^{-\lambda t_{xmt}} + \left. \frac{1}{\lambda} e^{-\lambda t} \right|_{t_{xmt}}^0 + t_{xmt}\lambda e^{-\lambda t_{xmt}} \]

\[ = \frac{1}{\lambda} (1-e^{-\lambda t_{xmt}})\]

\[\langle p(t)\rangle = \lambda \frac{1}{\lambda} (1-e^{-\lambda t_{xmt}})\]
\[\langle p(t)\rangle = (1-e^{-\lambda t_{xmt}})\]

The average synaptic conductance is then
\[\langle x \rangle = g_{max}(1-e^{-\lambda t_{xmt}})\]