Skip to content

Instantly share code, notes, and snippets.

@wrgeorge1983

wrgeorge1983/notebook export.md

Last active August 29, 2015 14:19
Show Gist options
  • Save wrgeorge1983/02b399f25fb0758e6329 to your computer and use it in GitHub Desktop.
Save wrgeorge1983/02b399f25fb0758e6329 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
notebook export.md 
import inspect
import pandas as pd

#Important Snippits

here's the sample data

sample_data = {'frame.time_epoch':[1429133053.239977000, 1429133053.245974000, 1429133053.250978000], 
               'frame.len': [60, 1414, 60], 
               'ip.src':['10.10.20.38', '20.20.5.97', '10.10.20.38'],
               'ip.dst':['20.20.5.97', '10.10.20.38', '20.20.5.97'],
               'tcp.stream':[0, 0, 0],
               'tcp.srcport':[59356, 80, 59356],
               'tcp.dstport':[80, 59356, 80]
              }

Default

doesn't reflect default ordering, because this comes from a dict and not the real output.
shouldn't matter.

df = pd.DataFrame.from_dict(sample_data)
df
frame.len frame.time_epoch ip.dst ip.src tcp.dstport tcp.srcport tcp.stream
0 60 1.429133e+09 20.20.5.97 10.10.20.38 80 59356 0
1 1414 1.429133e+09 10.10.20.38 20.20.5.97 59356 80 0
2 60 1.429133e+09 20.20.5.97 10.10.20.38 80 59356 0

Better Column Names

order is important, so we have to fudge this for sample data from dict.

# df = pd.read_table(output_file, names=['time','len','ip.src','ip.dst','stream','tcp.src', 'tcp.dst'], skiprows=1)

# simulate on sample_data

df.rename(columns={'frame.time_epoch': 'time',
                   'frame.len': 'len',
                   'ip.src': 'ip.src',
                   'ip.dst': 'ip.dst',
                   'tcp.stream': 'stream',
                   'tcp.srcport': 'tcpsrc',
                   'tcp.dstport': 'tcpdst'
                  }, inplace=True)
df
len time ip.dst ip.src tcpdst tcpsrc stream
0 60 1.429133e+09 20.20.5.97 10.10.20.38 80 59356 0
1 1414 1.429133e+09 10.10.20.38 20.20.5.97 59356 80 0
2 60 1.429133e+09 20.20.5.97 10.10.20.38 80 59356 0

Fix the Time

df.time = df.time.astype('datetime64[s]')
df
len time ip.dst ip.src tcpdst tcpsrc stream
0 60 2015-04-15 21:24:13 20.20.5.97 10.10.20.38 80 59356 0
1 1414 2015-04-15 21:24:13 10.10.20.38 20.20.5.97 59356 80 0
2 60 2015-04-15 21:24:13 20.20.5.97 10.10.20.38 80 59356 0

Filter For A Given Value

df[df.tcpdst == 80]
len time ip.dst ip.src tcpdst tcpsrc stream
0 60 2015-04-15 21:24:13 20.20.5.97 10.10.20.38 80 59356 0
2 60 2015-04-15 21:24:13 20.20.5.97 10.10.20.38 80 59356 0

Filter For Not In A Set Of Given Values

Mostly useless on 3 rows of data, but it works.

df[[True if x not in [1, 2, 3, 80, 145, 141] else False for x in df.tcpdst]]
len time ip.dst ip.src tcpdst tcpsrc stream
1 1414 2015-04-15 21:24:13 10.10.20.38 20.20.5.97 59356 80 0

More readable...

filtered_ports = [1, 2, 3, 80, 145, 141]
row_filter = [True if port not in filtered_ports else False for port in df.tcpdst]
df[row_filter]
len time ip.dst ip.src tcpdst tcpsrc stream
1 1414 2015-04-15 21:24:13 10.10.20.38 20.20.5.97 59356 80 0

3 Rows Isn't Cutting It

from the other notebook:
df2.head(100).to_json(orient='split')

json_from_other_nb = '{"time":{"0":1429133053239977,"1":1429133053245974,"2":1429133053250977,"3":1429133053254975,"4":1429133053254975,"5":1429133053260971,"6":1429133053265977,"7":1429133053265977,"8":1429133053269975,"9":1429133053274979,"10":1429133053274979,"11":1429133053279984,"12":1429133053284973,"13":1429133053288971,"14":1429133053297973,"15":1429133053297973,"16":1429133053297973,"17":1429133053302978,"18":1429133053306975,"19":1429133053311980,"20":1429133053315977,"21":1429133053320982,"22":1429133053320982,"23":1429133053327970,"24":1429133053331968,"25":1429133053335981,"26":1429133053335981,"27":1429133053340970,"28":1429133053344968,"29":1429133053344968,"30":1429133053348980,"31":1429133053353970,"32":1429133053357967,"33":1429133053362972,"34":1429133053366970,"35":1429133053366970,"36":1429133053366970,"37":1429133053373973,"38":1429133053377971,"39":1429133053377971,"40":1429133053382975,"41":1429133053387965,"42":1429133053391978,"43":1429133053396967,"44":1429133053400964,"45":1429133053404977,"46":1429133053409967,"47":1429133053409967,"48":1429133053415963,"49":1429133053420968,"50":1429133053424965,"51":1429133053429970,"52":1429133053433968,"53":1429133053438972,"54":1429133053442970,"55":1429133053442970,"56":1429133053446967,"57":1429133053455970,"58":1429133053460974,"59":1429133053464972,"60":1429133053469961,"61":1429133053473974,"62":1429133053478963,"63":1429133053478963,"64":1429133053484960,"65":1429133053488973,"66":1429133053492970,"67":1429133053492970,"68":1429133053497960,"69":1429133053501972,"70":1429133053501972,"71":1429133053509968,"72":1429133053515964,"73":1429133053519962,"74":1429133053525958,"75":1429133053529970,"76":1429133053534960,"77":1429133053534960,"78":1429133053538958,"79":1429133053543962,"80":1429133053543962,"81":1429133053547960,"82":1429133053551958,"83":1429133053551958,"84":1429133053557969,"85":1429133053566956,"86":1429133053575958,"87":1429133053575958,"88":1429133053579956,"89":1429133053579956,"90":1429133053584961,"91":1429133053588958,"92":1429133053588958,"93":1429133053588958,"94":1429133053598968,"95":1429133053602965,"96":1429133053606963,"97":1429133053612959,"98":1429133053616957,"99":1429133053616957},"len":{"0":60,"1":1414,"2":60,"3":1414,"4":1414,"5":1414,"6":60,"7":1414,"8":1414,"9":1414,"10":60,"11":1414,"12":1414,"13":60,"14":1414,"15":1414,"16":60,"17":1414,"18":1414,"19":60,"20":1414,"21":1414,"22":60,"23":1414,"24":1414,"25":60,"26":1414,"27":1414,"28":1414,"29":60,"30":1414,"31":60,"32":1414,"33":1414,"34":60,"35":1414,"36":60,"37":1414,"38":1414,"39":60,"40":1414,"41":60,"42":1414,"43":1414,"44":1414,"45":60,"46":1414,"47":1414,"48":60,"49":1414,"50":1414,"51":1414,"52":1414,"53":60,"54":1414,"55":1414,"56":60,"57":60,"58":1414,"59":1414,"60":1414,"61":1414,"62":60,"63":1414,"64":1414,"65":60,"66":1414,"67":1414,"68":60,"69":1414,"70":1414,"71":1414,"72":1414,"73":60,"74":60,"75":1414,"76":1414,"77":1414,"78":60,"79":1414,"80":1414,"81":60,"82":1414,"83":1414,"84":60,"85":1414,"86":1414,"87":1414,"88":60,"89":1414,"90":1414,"91":60,"92":1414,"93":1414,"94":1414,"95":1414,"96":60,"97":1414,"98":1414,"99":60},"ip.src":{"0":"10.10.20.38","1":"20.20.5.97","2":"10.10.20.38","3":"20.20.5.97","4":"20.20.5.97","5":"20.20.5.97","6":"10.10.20.38","7":"20.20.5.97","8":"20.20.5.97","9":"20.20.5.97","10":"10.10.20.38","11":"20.20.5.97","12":"20.20.5.97","13":"10.10.20.38","14":"20.20.5.97","15":"20.20.5.97","16":"10.10.20.38","17":"20.20.5.97","18":"20.20.5.97","19":"10.10.20.38","20":"20.20.5.97","21":"20.20.5.97","22":"10.10.20.38","23":"20.20.5.97","24":"20.20.5.97","25":"10.10.20.38","26":"20.20.5.97","27":"20.20.5.97","28":"20.20.5.97","29":"10.10.20.38","30":"20.20.5.97","31":"10.10.20.38","32":"20.20.5.97","33":"20.20.5.97","34":"10.10.20.38","35":"20.20.5.97","36":"10.10.104.82","37":"20.20.5.97","38":"20.20.5.97","39":"10.10.20.38","40":"20.20.5.97","41":"10.10.20.38","42":"20.20.5.97","43":"20.20.5.97","44":"20.20.5.97","45":"10.10.20.38","46":"20.20.5.97","47":"20.20.5.97","48":"10.10.20.38","49":"20.20.5.97","50":"20.20.5.97","51":"20.20.5.97","52":"20.20.5.97","53":"10.10.20.38","54":"20.20.5.97","55":"20.20.5.97","56":"10.10.20.38","57":"10.10.20.38","58":"20.20.5.97","59":"20.20.5.97","60":"20.20.5.97","61":"20.20.5.97","62":"10.10.20.38","63":"20.20.5.97","64":"20.20.5.97","65":"10.10.20.38","66":"20.20.5.97","67":"20.20.5.97","68":"10.10.20.38","69":"20.20.5.97","70":"20.20.5.97","71":"20.20.5.97","72":"20.20.5.97","73":"10.10.20.38","74":"10.10.20.38","75":"20.20.5.97","76":"20.20.5.97","77":"20.20.5.97","78":"10.10.20.38","79":"20.20.5.97","80":"20.20.5.97","81":"10.10.20.38","82":"20.20.5.97","83":"20.20.5.97","84":"10.10.20.38","85":"20.20.5.97","86":"20.20.5.97","87":"20.20.5.97","88":"10.10.20.38","89":"20.20.5.97","90":"20.20.5.97","91":"10.10.20.38","92":"20.20.5.97","93":"20.20.5.97","94":"20.20.5.97","95":"20.20.5.97","96":"10.10.20.38","97":"20.20.5.97","98":"20.20.5.97","99":"10.10.20.38"},"ip.dst":{"0":"20.20.5.97","1":"10.10.20.38","2":"20.20.5.97","3":"10.10.20.38","4":"10.10.20.38","5":"10.10.20.38","6":"20.20.5.97","7":"10.10.20.38","8":"10.10.20.38","9":"10.10.20.38","10":"20.20.5.97","11":"10.10.20.38","12":"10.10.20.38","13":"20.20.5.97","14":"10.10.20.38","15":"10.10.20.38","16":"20.20.5.97","17":"10.10.20.38","18":"10.10.20.38","19":"20.20.5.97","20":"10.10.20.38","21":"10.10.20.38","22":"20.20.5.97","23":"10.10.20.38","24":"10.10.20.38","25":"20.20.5.97","26":"10.10.20.38","27":"10.10.20.38","28":"10.10.20.38","29":"20.20.5.97","30":"10.10.20.38","31":"20.20.5.97","32":"10.10.20.38","33":"10.10.20.38","34":"20.20.5.97","35":"10.10.20.38","36":"10.10.20.25","37":"10.10.20.38","38":"10.10.20.38","39":"20.20.5.97","40":"10.10.20.38","41":"20.20.5.97","42":"10.10.20.38","43":"10.10.20.38","44":"10.10.20.38","45":"20.20.5.97","46":"10.10.20.38","47":"10.10.20.38","48":"20.20.5.97","49":"10.10.20.38","50":"10.10.20.38","51":"10.10.20.38","52":"10.10.20.38","53":"20.20.5.97","54":"10.10.20.38","55":"10.10.20.38","56":"20.20.5.97","57":"20.20.5.97","58":"10.10.20.38","59":"10.10.20.38","60":"10.10.20.38","61":"10.10.20.38","62":"20.20.5.97","63":"10.10.20.38","64":"10.10.20.38","65":"20.20.5.97","66":"10.10.20.38","67":"10.10.20.38","68":"20.20.5.97","69":"10.10.20.38","70":"10.10.20.38","71":"10.10.20.38","72":"10.10.20.38","73":"20.20.5.97","74":"20.20.5.97","75":"10.10.20.38","76":"10.10.20.38","77":"10.10.20.38","78":"20.20.5.97","79":"10.10.20.38","80":"10.10.20.38","81":"20.20.5.97","82":"10.10.20.38","83":"10.10.20.38","84":"20.20.5.97","85":"10.10.20.38","86":"10.10.20.38","87":"10.10.20.38","88":"20.20.5.97","89":"10.10.20.38","90":"10.10.20.38","91":"20.20.5.97","92":"10.10.20.38","93":"10.10.20.38","94":"10.10.20.38","95":"10.10.20.38","96":"20.20.5.97","97":"10.10.20.38","98":"10.10.20.38","99":"20.20.5.97"},"stream":{"0":0,"1":0,"2":0,"3":0,"4":0,"5":0,"6":0,"7":0,"8":0,"9":0,"10":0,"11":0,"12":0,"13":0,"14":0,"15":0,"16":0,"17":0,"18":0,"19":0,"20":0,"21":0,"22":0,"23":0,"24":0,"25":0,"26":0,"27":0,"28":0,"29":0,"30":0,"31":0,"32":0,"33":0,"34":0,"35":0,"36":1,"37":0,"38":0,"39":0,"40":0,"41":0,"42":0,"43":0,"44":0,"45":0,"46":0,"47":0,"48":0,"49":0,"50":0,"51":0,"52":0,"53":0,"54":0,"55":0,"56":0,"57":0,"58":0,"59":0,"60":0,"61":0,"62":0,"63":0,"64":0,"65":0,"66":0,"67":0,"68":0,"69":0,"70":0,"71":0,"72":0,"73":0,"74":0,"75":0,"76":0,"77":0,"78":0,"79":0,"80":0,"81":0,"82":0,"83":0,"84":0,"85":0,"86":0,"87":0,"88":0,"89":0,"90":0,"91":0,"92":0,"93":0,"94":0,"95":0,"96":0,"97":0,"98":0,"99":0},"tcp.src":{"0":59356,"1":80,"2":59356,"3":80,"4":80,"5":80,"6":59356,"7":80,"8":80,"9":80,"10":59356,"11":80,"12":80,"13":59356,"14":80,"15":80,"16":59356,"17":80,"18":80,"19":59356,"20":80,"21":80,"22":59356,"23":80,"24":80,"25":59356,"26":80,"27":80,"28":80,"29":59356,"30":80,"31":59356,"32":80,"33":80,"34":59356,"35":80,"36":50652,"37":80,"38":80,"39":59356,"40":80,"41":59356,"42":80,"43":80,"44":80,"45":59356,"46":80,"47":80,"48":59356,"49":80,"50":80,"51":80,"52":80,"53":59356,"54":80,"55":80,"56":59356,"57":59356,"58":80,"59":80,"60":80,"61":80,"62":59356,"63":80,"64":80,"65":59356,"66":80,"67":80,"68":59356,"69":80,"70":80,"71":80,"72":80,"73":59356,"74":59356,"75":80,"76":80,"77":80,"78":59356,"79":80,"80":80,"81":59356,"82":80,"83":80,"84":59356,"85":80,"86":80,"87":80,"88":59356,"89":80,"90":80,"91":59356,"92":80,"93":80,"94":80,"95":80,"96":59356,"97":80,"98":80,"99":59356},"tcp.dst":{"0":80,"1":59356,"2":80,"3":59356,"4":59356,"5":59356,"6":80,"7":59356,"8":59356,"9":59356,"10":80,"11":59356,"12":59356,"13":80,"14":59356,"15":59356,"16":80,"17":59356,"18":59356,"19":80,"20":59356,"21":59356,"22":80,"23":59356,"24":59356,"25":80,"26":59356,"27":59356,"28":59356,"29":80,"30":59356,"31":80,"32":59356,"33":59356,"34":80,"35":59356,"36":3389,"37":59356,"38":59356,"39":80,"40":59356,"41":80,"42":59356,"43":59356,"44":59356,"45":80,"46":59356,"47":59356,"48":80,"49":59356,"50":59356,"51":59356,"52":59356,"53":80,"54":59356,"55":59356,"56":80,"57":80,"58":59356,"59":59356,"60":59356,"61":59356,"62":80,"63":59356,"64":59356,"65":80,"66":59356,"67":59356,"68":80,"69":59356,"70":59356,"71":59356,"72":59356,"73":80,"74":80,"75":59356,"76":59356,"77":59356,"78":80,"79":59356,"80":59356,"81":80,"82":59356,"83":59356,"84":80,"85":59356,"86":59356,"87":59356,"88":80,"89":59356,"90":59356,"91":80,"92":59356,"93":59356,"94":59356,"95":59356,"96":80,"97":59356,"98":59356,"99":80}}'
print(json_from_other_nb)

{"time":{"0":1429133053239977,"1":1429133053245974,"2":1429133053250977,"3":1429133053254975,"4":1429133053254975,"5":1429133053260971,"6":1429133053265977,"7":1429133053265977,"8":1429133053269975,"9":1429133053274979,"10":1429133053274979,"11":1429133053279984,"12":1429133053284973,"13":1429133053288971,"14":1429133053297973,"15":1429133053297973,"16":1429133053297973,"17":1429133053302978,"18":1429133053306975,"19":1429133053311980,"20":1429133053315977,"21":1429133053320982,"22":1429133053320982,"23":1429133053327970,"24":1429133053331968,"25":1429133053335981,"26":1429133053335981,"27":1429133053340970,"28":1429133053344968,"29":1429133053344968,"30":1429133053348980,"31":1429133053353970,"32":1429133053357967,"33":1429133053362972,"34":1429133053366970,"35":1429133053366970,"36":1429133053366970,"37":1429133053373973,"38":1429133053377971,"39":1429133053377971,"40":1429133053382975,"41":1429133053387965,"42":1429133053391978,"43":1429133053396967,"44":1429133053400964,"45":1429133053404977,"46":1429133053409967,"47":1429133053409967,"48":1429133053415963,"49":1429133053420968,"50":1429133053424965,"51":1429133053429970,"52":1429133053433968,"53":1429133053438972,"54":1429133053442970,"55":1429133053442970,"56":1429133053446967,"57":1429133053455970,"58":1429133053460974,"59":1429133053464972,"60":1429133053469961,"61":1429133053473974,"62":1429133053478963,"63":1429133053478963,"64":1429133053484960,"65":1429133053488973,"66":1429133053492970,"67":1429133053492970,"68":1429133053497960,"69":1429133053501972,"70":1429133053501972,"71":1429133053509968,"72":1429133053515964,"73":1429133053519962,"74":1429133053525958,"75":1429133053529970,"76":1429133053534960,"77":1429133053534960,"78":1429133053538958,"79":1429133053543962,"80":1429133053543962,"81":1429133053547960,"82":1429133053551958,"83":1429133053551958,"84":1429133053557969,"85":1429133053566956,"86":1429133053575958,"87":1429133053575958,"88":1429133053579956,"89":1429133053579956,"90":1429133053584961,"91":1429133053588958,"92":1429133053588958,"93":1429133053588958,"94":1429133053598968,"95":1429133053602965,"96":1429133053606963,"97":1429133053612959,"98":1429133053616957,"99":1429133053616957},"len":{"0":60,"1":1414,"2":60,"3":1414,"4":1414,"5":1414,"6":60,"7":1414,"8":1414,"9":1414,"10":60,"11":1414,"12":1414,"13":60,"14":1414,"15":1414,"16":60,"17":1414,"18":1414,"19":60,"20":1414,"21":1414,"22":60,"23":1414,"24":1414,"25":60,"26":1414,"27":1414,"28":1414,"29":60,"30":1414,"31":60,"32":1414,"33":1414,"34":60,"35":1414,"36":60,"37":1414,"38":1414,"39":60,"40":1414,"41":60,"42":1414,"43":1414,"44":1414,"45":60,"46":1414,"47":1414,"48":60,"49":1414,"50":1414,"51":1414,"52":1414,"53":60,"54":1414,"55":1414,"56":60,"57":60,"58":1414,"59":1414,"60":1414,"61":1414,"62":60,"63":1414,"64":1414,"65":60,"66":1414,"67":1414,"68":60,"69":1414,"70":1414,"71":1414,"72":1414,"73":60,"74":60,"75":1414,"76":1414,"77":1414,"78":60,"79":1414,"80":1414,"81":60,"82":1414,"83":1414,"84":60,"85":1414,"86":1414,"87":1414,"88":60,"89":1414,"90":1414,"91":60,"92":1414,"93":1414,"94":1414,"95":1414,"96":60,"97":1414,"98":1414,"99":60},"ip.src":{"0":"10.10.20.38","1":"20.20.5.97","2":"10.10.20.38","3":"20.20.5.97","4":"20.20.5.97","5":"20.20.5.97","6":"10.10.20.38","7":"20.20.5.97","8":"20.20.5.97","9":"20.20.5.97","10":"10.10.20.38","11":"20.20.5.97","12":"20.20.5.97","13":"10.10.20.38","14":"20.20.5.97","15":"20.20.5.97","16":"10.10.20.38","17":"20.20.5.97","18":"20.20.5.97","19":"10.10.20.38","20":"20.20.5.97","21":"20.20.5.97","22":"10.10.20.38","23":"20.20.5.97","24":"20.20.5.97","25":"10.10.20.38","26":"20.20.5.97","27":"20.20.5.97","28":"20.20.5.97","29":"10.10.20.38","30":"20.20.5.97","31":"10.10.20.38","32":"20.20.5.97","33":"20.20.5.97","34":"10.10.20.38","35":"20.20.5.97","36":"10.10.104.82","37":"20.20.5.97","38":"20.20.5.97","39":"10.10.20.38","40":"20.20.5.97","41":"10.10.20.38","42":"20.20.5.97","43":"20.20.5.97","44":"20.20.5.97","45":"10.10.20.38","46":"20.20.5.97","47":"20.20.5.97","48":"10.10.20.38","49":"20.20.5.97","50":"20.20.5.97","51":"20.20.5.97","52":"20.20.5.97","53":"10.10.20.38","54":"20.20.5.97","55":"20.20.5.97","56":"10.10.20.38","57":"10.10.20.38","58":"20.20.5.97","59":"20.20.5.97","60":"20.20.5.97","61":"20.20.5.97","62":"10.10.20.38","63":"20.20.5.97","64":"20.20.5.97","65":"10.10.20.38","66":"20.20.5.97","67":"20.20.5.97","68":"10.10.20.38","69":"20.20.5.97","70":"20.20.5.97","71":"20.20.5.97","72":"20.20.5.97","73":"10.10.20.38","74":"10.10.20.38","75":"20.20.5.97","76":"20.20.5.97","77":"20.20.5.97","78":"10.10.20.38","79":"20.20.5.97","80":"20.20.5.97","81":"10.10.20.38","82":"20.20.5.97","83":"20.20.5.97","84":"10.10.20.38","85":"20.20.5.97","86":"20.20.5.97","87":"20.20.5.97","88":"10.10.20.38","89":"20.20.5.97","90":"20.20.5.97","91":"10.10.20.38","92":"20.20.5.97","93":"20.20.5.97","94":"20.20.5.97","95":"20.20.5.97","96":"10.10.20.38","97":"20.20.5.97","98":"20.20.5.97","99":"10.10.20.38"},"ip.dst":{"0":"20.20.5.97","1":"10.10.20.38","2":"20.20.5.97","3":"10.10.20.38","4":"10.10.20.38","5":"10.10.20.38","6":"20.20.5.97","7":"10.10.20.38","8":"10.10.20.38","9":"10.10.20.38","10":"20.20.5.97","11":"10.10.20.38","12":"10.10.20.38","13":"20.20.5.97","14":"10.10.20.38","15":"10.10.20.38","16":"20.20.5.97","17":"10.10.20.38","18":"10.10.20.38","19":"20.20.5.97","20":"10.10.20.38","21":"10.10.20.38","22":"20.20.5.97","23":"10.10.20.38","24":"10.10.20.38","25":"20.20.5.97","26":"10.10.20.38","27":"10.10.20.38","28":"10.10.20.38","29":"20.20.5.97","30":"10.10.20.38","31":"20.20.5.97","32":"10.10.20.38","33":"10.10.20.38","34":"20.20.5.97","35":"10.10.20.38","36":"10.10.20.25","37":"10.10.20.38","38":"10.10.20.38","39":"20.20.5.97","40":"10.10.20.38","41":"20.20.5.97","42":"10.10.20.38","43":"10.10.20.38","44":"10.10.20.38","45":"20.20.5.97","46":"10.10.20.38","47":"10.10.20.38","48":"20.20.5.97","49":"10.10.20.38","50":"10.10.20.38","51":"10.10.20.38","52":"10.10.20.38","53":"20.20.5.97","54":"10.10.20.38","55":"10.10.20.38","56":"20.20.5.97","57":"20.20.5.97","58":"10.10.20.38","59":"10.10.20.38","60":"10.10.20.38","61":"10.10.20.38","62":"20.20.5.97","63":"10.10.20.38","64":"10.10.20.38","65":"20.20.5.97","66":"10.10.20.38","67":"10.10.20.38","68":"20.20.5.97","69":"10.10.20.38","70":"10.10.20.38","71":"10.10.20.38","72":"10.10.20.38","73":"20.20.5.97","74":"20.20.5.97","75":"10.10.20.38","76":"10.10.20.38","77":"10.10.20.38","78":"20.20.5.97","79":"10.10.20.38","80":"10.10.20.38","81":"20.20.5.97","82":"10.10.20.38","83":"10.10.20.38","84":"20.20.5.97","85":"10.10.20.38","86":"10.10.20.38","87":"10.10.20.38","88":"20.20.5.97","89":"10.10.20.38","90":"10.10.20.38","91":"20.20.5.97","92":"10.10.20.38","93":"10.10.20.38","94":"10.10.20.38","95":"10.10.20.38","96":"20.20.5.97","97":"10.10.20.38","98":"10.10.20.38","99":"20.20.5.97"},"stream":{"0":0,"1":0,"2":0,"3":0,"4":0,"5":0,"6":0,"7":0,"8":0,"9":0,"10":0,"11":0,"12":0,"13":0,"14":0,"15":0,"16":0,"17":0,"18":0,"19":0,"20":0,"21":0,"22":0,"23":0,"24":0,"25":0,"26":0,"27":0,"28":0,"29":0,"30":0,"31":0,"32":0,"33":0,"34":0,"35":0,"36":1,"37":0,"38":0,"39":0,"40":0,"41":0,"42":0,"43":0,"44":0,"45":0,"46":0,"47":0,"48":0,"49":0,"50":0,"51":0,"52":0,"53":0,"54":0,"55":0,"56":0,"57":0,"58":0,"59":0,"60":0,"61":0,"62":0,"63":0,"64":0,"65":0,"66":0,"67":0,"68":0,"69":0,"70":0,"71":0,"72":0,"73":0,"74":0,"75":0,"76":0,"77":0,"78":0,"79":0,"80":0,"81":0,"82":0,"83":0,"84":0,"85":0,"86":0,"87":0,"88":0,"89":0,"90":0,"91":0,"92":0,"93":0,"94":0,"95":0,"96":0,"97":0,"98":0,"99":0},"tcp.src":{"0":59356,"1":80,"2":59356,"3":80,"4":80,"5":80,"6":59356,"7":80,"8":80,"9":80,"10":59356,"11":80,"12":80,"13":59356,"14":80,"15":80,"16":59356,"17":80,"18":80,"19":59356,"20":80,"21":80,"22":59356,"23":80,"24":80,"25":59356,"26":80,"27":80,"28":80,"29":59356,"30":80,"31":59356,"32":80,"33":80,"34":59356,"35":80,"36":50652,"37":80,"38":80,"39":59356,"40":80,"41":59356,"42":80,"43":80,"44":80,"45":59356,"46":80,"47":80,"48":59356,"49":80,"50":80,"51":80,"52":80,"53":59356,"54":80,"55":80,"56":59356,"57":59356,"58":80,"59":80,"60":80,"61":80,"62":59356,"63":80,"64":80,"65":59356,"66":80,"67":80,"68":59356,"69":80,"70":80,"71":80,"72":80,"73":59356,"74":59356,"75":80,"76":80,"77":80,"78":59356,"79":80,"80":80,"81":59356,"82":80,"83":80,"84":59356,"85":80,"86":80,"87":80,"88":59356,"89":80,"90":80,"91":59356,"92":80,"93":80,"94":80,"95":80,"96":59356,"97":80,"98":80,"99":59356},"tcp.dst":{"0":80,"1":59356,"2":80,"3":59356,"4":59356,"5":59356,"6":80,"7":59356,"8":59356,"9":59356,"10":80,"11":59356,"12":59356,"13":80,"14":59356,"15":59356,"16":80,"17":59356,"18":59356,"19":80,"20":59356,"21":59356,"22":80,"23":59356,"24":59356,"25":80,"26":59356,"27":59356,"28":59356,"29":80,"30":59356,"31":80,"32":59356,"33":59356,"34":80,"35":59356,"36":3389,"37":59356,"38":59356,"39":80,"40":59356,"41":80,"42":59356,"43":59356,"44":59356,"45":80,"46":59356,"47":59356,"48":80,"49":59356,"50":59356,"51":59356,"52":59356,"53":80,"54":59356,"55":59356,"56":80,"57":80,"58":59356,"59":59356,"60":59356,"61":59356,"62":80,"63":59356,"64":59356,"65":80,"66":59356,"67":59356,"68":80,"69":59356,"70":59356,"71":59356,"72":59356,"73":80,"74":80,"75":59356,"76":59356,"77":59356,"78":80,"79":59356,"80":59356,"81":80,"82":59356,"83":59356,"84":80,"85":59356,"86":59356,"87":59356,"88":80,"89":59356,"90":59356,"91":80,"92":59356,"93":59356,"94":59356,"95":59356,"96":80,"97":59356,"98":59356,"99":80}}



df2 = pd.io.json.read_json(json_from_other_nb)
df2.head(10)
ip.dst ip.src len stream tcp.dst tcp.src time
0 20.20.5.97 10.10.20.38 60 0 80 59356 1429133053239977
1 10.10.20.38 20.20.5.97 1414 0 59356 80 1429133053245974
10 20.20.5.97 10.10.20.38 60 0 80 59356 1429133053274979
11 10.10.20.38 20.20.5.97 1414 0 59356 80 1429133053279984
12 10.10.20.38 20.20.5.97 1414 0 59356 80 1429133053284973
13 20.20.5.97 10.10.20.38 60 0 80 59356 1429133053288971
14 10.10.20.38 20.20.5.97 1414 0 59356 80 1429133053297973
15 10.10.20.38 20.20.5.97 1414 0 59356 80 1429133053297973
16 20.20.5.97 10.10.20.38 60 0 80 59356 1429133053297973
17 10.10.20.38 20.20.5.97 1414 0 59356 80 1429133053302978

Magic!

but time is jacked up...

df2.time = pd.to_datetime(df2.time*(10**-6), unit='s')
df2
ip.dst ip.src len stream tcp.dst tcp.src time
0 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.239977
1 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.245974
10 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.274979
11 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.279984
12 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.284973
13 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.288971
14 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.297973
15 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.297973
16 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.297973
17 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.302978
18 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.306975
19 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.311980
2 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.250977
20 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.315977
21 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.320982
22 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.320982
23 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.327970
24 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.331968
25 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.335981
26 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.335981
27 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.340970
28 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.344968
29 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.344968
3 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.254975
30 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.348980
31 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.353970
32 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.357967
33 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.362972
34 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.366970
35 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.366970
... ... ... ... ... ... ... ...
72 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.515964
73 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.519962
74 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.525958
75 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.529970
76 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.534960
77 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.534960
78 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.538958
79 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.543962
8 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.269975
80 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.543962
81 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.547960
82 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.551958
83 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.551958
84 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.557969
85 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.566956
86 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.575958
87 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.575958
88 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.579956
89 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.579956
9 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.274979
90 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.584961
91 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.588958
92 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.588958
93 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.588958
94 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.598968
95 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.602965
96 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.606963
97 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.612959
98 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.616957
99 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.616957

100 rows × 7 columns

df2.sort_index(inplace=True)
df2.head(10)
ip.dst ip.src len stream tcp.dst tcp.src time
0 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.239977
1 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.245974
2 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.250977
3 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.254975
4 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.254975
5 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.260971
6 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.265977
7 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.265977
8 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.269975
9 10.10.20.38 20.20.5.97 1414 0 59356 80 2015-04-15 21:24:13.274979 
[stream for stream in df2.stream if stream != 0]




[1]
So we only have one frame that isn't in stream 0,

whatever, we can still pivot it and try to extract flows

Using MultiIndexing:

#mi = df2.set_index('stream').set_index('ip.src', append=True)
#mi
#loc = mi.loc[0,'10.10.20.38']  only works for actual multi-indexing.  not needed otherwise
#loc

Using Filtering (preferred, so far):

df2_filter = (df2.stream == 0) & (df2['ip.src'] == '10.10.20.38')
loc = df2[df2_filter]
loc
ip.dst ip.src len stream tcp.dst tcp.src time
0 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.239977
2 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.250977
6 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.265977
10 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.274979
13 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.288971
16 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.297973
19 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.311980
22 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.320982
25 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.335981
29 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.344968
31 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.353970
34 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.366970
39 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.377971
41 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.387965
45 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.404977
48 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.415963
53 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.438972
56 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.446967
57 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.455970
62 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.478963
65 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.488973
68 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.497960
73 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.519962
74 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.525958
78 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.538958
81 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.547960
84 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.557969
88 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.579956
91 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.588958
96 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.606963
99 20.20.5.97 10.10.20.38 60 0 80 59356 2015-04-15 21:24:13.616957

This Works For MultiIndexing As Well As Normal Filtering

flow_frames = loc[['len', 'time']]
# # These lines are only needed if you used multi-indexing
# flow_frames = flow_frames.reset_index()
# flow_frames.drop(['stream','ip.src'], axis=1, inplace=True)
flow_frames
len time
0 60 2015-04-15 21:24:13.239977
2 60 2015-04-15 21:24:13.250977
6 60 2015-04-15 21:24:13.265977
10 60 2015-04-15 21:24:13.274979
13 60 2015-04-15 21:24:13.288971
16 60 2015-04-15 21:24:13.297973
19 60 2015-04-15 21:24:13.311980
22 60 2015-04-15 21:24:13.320982
25 60 2015-04-15 21:24:13.335981
29 60 2015-04-15 21:24:13.344968
31 60 2015-04-15 21:24:13.353970
34 60 2015-04-15 21:24:13.366970
39 60 2015-04-15 21:24:13.377971
41 60 2015-04-15 21:24:13.387965
45 60 2015-04-15 21:24:13.404977
48 60 2015-04-15 21:24:13.415963
53 60 2015-04-15 21:24:13.438972
56 60 2015-04-15 21:24:13.446967
57 60 2015-04-15 21:24:13.455970
62 60 2015-04-15 21:24:13.478963
65 60 2015-04-15 21:24:13.488973
68 60 2015-04-15 21:24:13.497960
73 60 2015-04-15 21:24:13.519962
74 60 2015-04-15 21:24:13.525958
78 60 2015-04-15 21:24:13.538958
81 60 2015-04-15 21:24:13.547960
84 60 2015-04-15 21:24:13.557969
88 60 2015-04-15 21:24:13.579956
91 60 2015-04-15 21:24:13.588958
96 60 2015-04-15 21:24:13.606963
99 60 2015-04-15 21:24:13.616957

Biggest hurdle right now is getting one of these for each flow, and attaching it to a "flow record" that holds the rest of the relevant data for each flow.

index p.dst ip.src stream tcp.dst tcp.src frame_df
0 20.20.5.97 10.10.20.38 60 80 59356 df({...})
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment