0:00:09hello guys
0:00:11is under very and grady leading the announces admin team
0:00:16and this for very
0:00:19is correctly
0:00:21i notice a sub men
0:00:25will be telling you about how we make up twenty min age
0:00:48okay so
0:00:51a bit will be telling you about how we men age our influx store
0:00:56out of site softer side
0:00:58what will be the future plans of the infrastructure
0:01:03a lot of more stuff
0:01:05so
0:01:07i live at the word to three
0:01:10for the first just lights
0:01:14okay so what we do
0:01:16so we met can so seventeen so we have a number of roles we have
0:01:24been making infrastructure so that means number web services own about things like that of
0:01:33course we have get
0:01:36those kind of things
0:01:38our main job to make sure everything is stable and you know and running smoothly
0:01:46right so and we also
0:01:50try to follow but every single team in the rubber in and copyright is to
0:01:55have a problem so it something goes down or we hear about is the right
0:02:01we are jobs trying to fix it
0:02:04we also coordinated our hardware sponsors for
0:02:10anything related you know for is red hat
0:02:14we work with red hat and so forth
0:02:18a big ever
0:02:21was our sponsors i would doubt that about the next slide well our sponsors are
0:02:28currently read a canonical down they are gonna state university
0:02:33the open source lap
0:02:36so do the right down in fact sort or has been really helpful with us
0:02:42many times and i'm really thankful to them
0:02:47and you can switch the
0:02:50so
0:02:52i would do we communicate with each other
0:02:56we teach sys admin
0:02:58we do have
0:03:00a channel on that your see that's going on that are which is sys admin
0:03:06we do have a public mailing list and we all set up private one where
0:03:11we receive all the programs all the proper grounds
0:03:15domain renewals
0:03:16also we discuss and it in private
0:03:21like private matters
0:03:22and we also set up a lot of new services for better can users and
0:03:28developers to know exactly what's happening when i went for accept or
0:03:34and these services are
0:03:37start does not than on that are and we have a we have a little
0:03:42then we are
0:03:43that's three will we show you
0:03:48okay
0:03:49so we have a good bunch of services
0:03:52and indicators something is wrong
0:03:56you connect to the service or you can even keep this page on your are
0:04:02that's filtered
0:04:06and you get each update when something goes wrong so let's say that something's wrong
0:04:12with other
0:04:14but we sure are red
0:04:16that buttons saying that there is a problem and we have quality checking what's going
0:04:23on there
0:04:24so keep this page bookmark every time and before asking us on the sys admin
0:04:29channel get yourself on this package and you know what we don't really know what's
0:04:35going on
0:04:46and
0:04:47okay
0:04:52then
0:04:53we created
0:04:55i'm not remaining liz i'm sorry for that but
0:04:59when i'm sending my announcement made as i was sending them to desktop double is
0:05:05a foundational list for mission announce so they created in fact that are announce whether
0:05:10i'd be sending well my emails about what's going on only for accept or and
0:05:15i also created a public not just you if you connect to ideas but the
0:05:21number per week to username anonymous and plus or anonymous you see old the services
0:05:30we run and the status of them so
0:05:33it's even better than the start of the dog the them that or one because
0:05:39not yours is a real time service so it sends a checks to the machine
0:05:45if the czech was wrong it repulsed that if i there immediately so
0:05:51just i don't have a preview of that but you can check that on your
0:05:56or your machine stuff
0:05:59okay
0:06:01so
0:06:03let's talk about the artwork
0:06:06read a is kindly austin
0:06:09a lot of machine for us
0:06:12a total of twenty two machines
0:06:16both vector machine and physical machines at the phoenix
0:06:20on their things chewed at the centre
0:06:25then canonical exhausting well two machines none of them exhausting the guy that website and
0:06:31they're going on website
0:06:34and down the oregon open-source lot exhausting
0:06:40i wanna not be as box and
0:06:43the testing machines for that website themed that and vested you about on in stock
0:06:50we do not see lee use lee peered and capable
0:06:57and
0:06:59we actually have just eight physical machines which are but huge in terms of
0:07:06a around C P U and so we use a lot of
0:07:12we promotions
0:07:14and there is something special and you now we name our servers
0:07:21for example
0:07:23demolition that currently all's a going on that are it's called socket
0:07:29many other caulk on the box or create bar or something like that
0:07:35and the name is are taken for day G T K we just
0:07:40so if you will do a reverse D N S check on a rolls you'll
0:07:46find out some strange names don't worry because is that is that actually expected
0:07:53okay so let's go to the software
0:07:57i will in age our web servers
0:08:01we do have
0:08:03an apache server in front of H a proxy
0:08:09so we have
0:08:11should bucks as which acts L's front ends
0:08:15and the fact that can machines actually are
0:08:21E P less so they don't have a public E P and there couldn't you
0:08:25just accessible by this is something team
0:08:30and i'll does H a proxy works there is round probably in the innocent the
0:08:36N S entry so you're clean and will
0:08:41randomly get an E P and connect to one of the two products is
0:08:49and this is call fail over
0:08:52so i wrote boxes are set up enough a little remote
0:08:55and
0:08:56we use an apache in front of a chip proxy because
0:09:01we're ready back well the best
0:09:04to ask ourselves so if you to check well than on the non services well
0:09:10down are us
0:09:11which ssl right now
0:09:14and of a can machine are abused lee adjustable or only by a lot that
0:09:22can also something
0:09:26so time for free
0:09:31okay so on the software side we have a number services or actually more accurate
0:09:37way i'm kind of i'm going to describe what environment looks like so i'm not
0:09:42so machines everything is N F S mounted so users generally have a consisted you
0:09:48still are going well
0:09:50not users but systems as ministers on the whole network we have open P N
0:09:56for we she's posted outside the phoenix data centre for actually the private network so
0:10:04you actually have to dpmc axis these then the phoenix data centre and that's just
0:10:11right for security purposes we also have open L that
0:10:17for our managing central access over the good this is primarily how we do accounts
0:10:24i think it's not
0:10:27in those people know what that is finally we have or are buying for serving
0:10:33guinness
0:10:35i don't think there's much to say about that we have to name servers well
0:10:40to
0:10:43cashed us requests and then finally we have a gonna jabber server which book talk
0:10:50further and slides
0:10:54the next that we have this prospects and the spend virus cover
0:11:00to so jonas our mail server and of course we have some stuff to catch
0:11:06spam
0:11:08we germans were classes web platform for about that be that the minimum that gonna
0:11:16that word and black second another
0:11:19probably can sample are using more annoying
0:11:23and that's a and we also have
0:11:27okay nice extensions that can and other which is
0:11:31so i per class with an that was about but the by jasper
0:11:38and then we have you can that which is a nice note taking application
0:11:44that let's you
0:11:47it's text and data that whatever we been using extensively at the marketing team especially
0:11:53and using it extensively for taking notes
0:11:57for on like prayer meetings like honesty and so forth
0:12:03and now i've got it yes
0:12:07so i we manage ourselves
0:12:12we evenly use pop but we do have an internal proper repository manage we the
0:12:18get
0:12:19and each comment we do is than notified to all this is augmenting so i
0:12:25actually know was working on wall
0:12:28and all the team is aware that they shouldn't archer specific model because i'm doing
0:12:35some changes there
0:12:37i'll pop i'm not sure i know many of you blocks leno our puppet work
0:12:44no one
0:12:47so top it is a centralised management system
0:12:51so you keep your configuration files in one place
0:12:55and that one place is called puppet master
0:12:59you down synchronise you create model models
0:13:03and you synchronise your machines to a specific model so you are able to recreate
0:13:10an entire machine we just two clicks
0:13:13and by running a specific common so it's really and they to rebuild an entire
0:13:19virtual machine in the case for example you're operating from well five to wreck six
0:13:27we also keep are standard operating procedures on our we key because
0:13:33each softer women age is configured is i evilly customized so not this is something
0:13:40is are aware about our we did a custom on is that we key or
0:13:45like as a lot or adapt so this all is therefore do specific motivation you
0:13:52check the soap and you have all distractions you need to find out are you
0:13:58should probably men age a specific service that was set up by someone else
0:14:07so we come to the backups are you we don't have a back lit a
0:14:11machine
0:14:13and we manage our backups in a strange way i mean
0:14:18we were on a nightly us think around that sense well the doctor of all
0:14:24the service to a north carolina know when
0:14:29to an or carolina facility
0:14:32and from there
0:14:36we get all the backups on the tape on tape drives
0:14:41and
0:14:43when vote a little script
0:14:46for that i was going to ask and exclude list that means that we can
0:14:51exclude specific directories from that we of the for specifics five system and exclude
0:14:59i dunno cache files for example or flies or been mounts so we don't duplicate
0:15:05the data on the on the backup
0:15:10well the lots of us sink are then sent to a machine well and that
0:15:17machine generates i'm apart every day so we get the mail which does something like
0:15:26well the machines are fine all the backups one fine or something is wrong with
0:15:31the of with a specific machine so you should check what's going
0:15:36and this is called a tear apart and you should think we should dangling providing
0:15:42just very nice by transcript
0:15:46so i want to we do our monitoring we are easily using now i guess
0:15:53using not just with a lot of customisation
0:15:56and the main one is called czech and K
0:16:00second czech and K is a no additional software you can actually willing to not
0:16:07guess
0:16:08and you get
0:16:10and ten or more of more checks
0:16:14without doing anything special
0:16:16it does that for it just work out the box you just up to link
0:16:20it on when you're not just interface and configure a few things
0:16:26we also have
0:16:30i don't and those are really nice feature because
0:16:34sometimes apache can go down
0:16:38and
0:16:39for whatever reason or you just got that time out i don't and those are
0:16:44there for that
0:16:45and our that are those they work in fact they
0:16:50send a check to specific service
0:16:54and if the czech is critical it automatically or starts they H T P D
0:17:00demon
0:17:02so they are very and a and then we have P N P for now
0:17:07guess which is a C E mail are too
0:17:12i don't know to comedy or
0:17:15to collect D it creates a pair often service statistics over the time so we
0:17:21know
0:17:23a real i'm not then machines slowly is or how much ram is being used
0:17:30during the during that time
0:17:34so
0:17:37the future plans which answer many of your are interesting
0:17:43and many of you
0:17:46are complaining about bags a lot
0:17:49because our but still a sense is very old
0:17:55is version three don't for one of the neural is far but for but why
0:18:02that's great didn't come yet
0:18:04the normal as many extensions
0:18:07customise we customise of our but still a sense
0:18:12with many extensions that the we have written by a lot of readers because i
0:18:17don't know if
0:18:18some of you know marine
0:18:21and unfortunately is not contributing anymore and
0:18:27why talking with william
0:18:30the other day you offered in south to
0:18:34as many or of writing
0:18:37these extensions
0:18:40to the new but still a format so we can probably planned an upgrade of
0:18:46our but still are doing the next six months i guess it will probably take
0:18:52that we will probably have to take down the service for around two days i
0:18:56guess because the them
0:18:59the database you which is around twenty gigabytes and restoring the world down will take
0:19:06i wanted to six to forty eight hours i guess
0:19:11the next future plans is
0:19:15or a locating are machines at phoenix to currently our an actress so you switch
0:19:23the networks which we are currently using is the one of another rack
0:19:28and that's definitely something we have to fix and we are collaborating with the field
0:19:34are guys to phoenix to
0:19:37to have all the management consoles attached to the breach of the machines and so
0:19:43we can track down
0:19:44well the errors of the console of the physical machines
0:19:54our doesn't work we thought each machine also management console part
0:20:00you connect D is mush this us which this part to the natural speech
0:20:07you get then E P i username and a password
0:20:12then you got yourself into the V P N
0:20:17and you cannot assess their web interface that allows you to report the mission check
0:20:23the console rest are gonna should down the machine on song
0:20:29and get all the data else about the method man a vector of for example
0:20:32so we got all the details about the physical machine itself
0:20:38we also plan to remove shots from master
0:20:43because going to there are a lot or a lot of active accounts and
0:20:52well of them do have S H adjust to master
0:20:56and i filled is that this is not very good secreted speaking so will be
0:21:02removing assess to master and we'll be creating
0:21:06a script
0:21:08to do that
0:21:11for your locomotion and we have think about quitting hourly the web interface what you
0:21:17cannot indicate using your old up a sort and you can just then click on
0:21:23upload file you select your top are your trouble from your pc and you click
0:21:31on in style
0:21:33this we call death dp admin script from the machine and do the job for
0:21:39you so we finally got rid of
0:21:42of the shelves
0:21:45and really proud of the last season
0:21:50and thanks to about through is what did all the work
0:21:54and will be monitoring the work it not the norm to talk to get out
0:22:00and this without plan program i think applause festival of so when you comment to
0:22:06the to the repository
0:22:09it to we run a little nearer
0:22:12that will saying well the content of your
0:22:16of your model you have to get up
0:22:21but will be announcing that is in a better way with more get those in
0:22:26the near future i guess
0:22:28which will the instructions you may need
0:22:59it's just are coming or
0:23:03it's
0:23:13okay
0:23:15you know
0:23:18alright and the you want advantage is that we can track the activity around the
0:23:24particular project
0:23:26we just one click in we can see who's working what they're doing against master
0:23:30right so
0:23:33resistors and i i'm certain your
0:23:36get more details later cool
0:23:40well also given the fact that katie the are pretty bad this group shown i
0:23:47don't know a few months ago and the they didn't have the backups
0:23:53so we are also mirroring to make sure that we have to backups one sound
0:23:59to the to the external facility every night and one get up
0:24:07we did to time their guitar people because they've been really up for we does
0:24:11and open a about or setting design
0:24:18so actually
0:24:21okay so this part is really the fun part this is the part where we
0:24:25get tell you what you can use an and to help you to have cooperation
0:24:32the first one is me but which is a it lets you do meetings on
0:24:38I R C and what it nicely
0:24:42it it's a it's a software that most people don't know about which is why
0:24:45i'm discussing it here
0:24:48we got that's you if you want you want to start you just say like
0:24:52you tell be bopping gonna start the meeting and then you are able to have
0:24:56all your discussions lark you can actually have somebody has a leader so for it's
0:25:06really quite cool and
0:25:09mostly done you can just kinda and the meeting and then you automatically get lots
0:25:15published a on a web on the website now i don't have some complaints about
0:25:21this because a lot of people want the logs to be private
0:25:28and so that's what the feedback we've gotten and we are gonna look at how
0:25:34to do be lot on a per channel
0:25:37basis and so if we can be configured like say if you sort of meeting
0:25:43on like how you know right you can expect that's gonna be public whereas if
0:25:50we started one in marketing or you know some other channel then those would be
0:25:56private
0:25:57so that's one thing there
0:26:02D this light has a link to be documentation so i will encourage you to
0:26:08look through that use it and give us some feedback because i think it'll really
0:26:15help us out when trying to collaboration
0:26:21the next one is you think that a talk about this before
0:26:26we use is a lot
0:26:27and especially on the marketing team we did this
0:26:34constantly internet quite like assembly said we for princes we were putting in but
0:26:41we're doing summarise so each day and we've been using this it's
0:26:50i basically you can have three four people adding this
0:26:56this page and you can have chat and you also know who's type being white
0:27:02because a lot different colours it
0:27:05right we can always try to do a demo real quick we've had some feedback
0:27:10on this as well
0:27:13you know the one of the things that happen on this is that you only
0:27:17get one either bad and if you have any us either pad before a lot
0:27:23of people would like to use multiple because when you go to it you only
0:27:28get this one page and if somebody's already using it then you don't really know
0:27:33what to do it so you end up people and of going to pilot patterson
0:27:37last so we're looking at trying to create
0:27:43new ones on the fly so we don't do it before because we're worried about
0:27:49spam but we will try to
0:27:54doing experiments if it works and if you don't get spam right if we don't
0:27:59get spam then well we'll figure out what to do at that point
0:28:09and we had a whole session on this
0:28:13well
0:28:17but if you haven't been to if you didn't go to that chuck this is
0:28:21a this is really cool because this allows you to do document storage right so
0:28:27we document storage you can drag and drop whatever like open up this or any
0:28:34number text sometimes if you have graphics design things like that it's a nice place
0:28:41to put it
0:28:43without having to use google doctor some other or other some other proprietary stuff so
0:28:53the other cool thing is you can share that if you click on it and
0:28:58say share with you can share with everyone you could create a public url just
0:29:03like you can't are dark box and it's it works very well
0:29:10so that's another nice thing
0:29:14if you wanna use it
0:29:17to create about and we'll be happy to give you access has using all that
0:29:25much of the secure that
0:29:30what about like to encourage them to use it more
0:29:34so
0:29:39are you okay
0:29:47well then right need you put some screenshot so something like work in progress markup
0:29:52of say website or whatever or just like a simple stupid disposable thing right which
0:30:00store some services for but they are like i know how they are right so
0:30:06what it used to do is to run this in a temporary folder on my
0:30:10own server a but i'm very happy that it's a this thing excess because it
0:30:16also i lost thank you very quickly easily charing public files winners and yes this
0:30:22is what is looks like sorry so
0:30:24thank you for sending data
0:30:27and the guy course is relevant work an integration with you know right we'd gone
0:30:32online docking online
0:30:34services i want accounts are some reason i remember but reverie doing arrived are putting
0:30:43like it's gonna show up are gonna documents and so for so i was just
0:30:50wondering when it is you just going to be for national number for like the
0:30:54Y O the foundation members only yet so this is sort of a something that
0:31:00you get
0:31:01for our as a as a
0:31:06as a benefit so
0:31:09we should we should definitely do that i do understand that it's probably us too
0:31:14much from that if structure just to make it available to everybody but
0:31:20is
0:31:23i mean it's a service we expect you that you're gonna use it four gram
0:31:28related activities as opposed to you know if it was why then you know we
0:31:33would end up becoming a service provider right if it becomes too popular we might
0:31:38you might have to
0:31:40do something else because all that stuff takes place and you never know
0:31:48the on the other it is to maybe have a
0:31:51monthly payment yes
0:31:59that would be a good a good a good go to
0:32:22okay
0:32:27not the
0:32:29the main concern i have is about privacy because opening are instance to the world
0:32:37we sure
0:32:38provide some of the old documents saying
0:32:42our paul is our legal places the S
0:32:47so
0:33:09okay
0:33:12so i will like to take a few minutes to remember frown
0:33:19a great friend
0:33:20i work at we sat
0:33:23for mass
0:33:25i can tell to us actually on the for the way facts or and it
0:33:30was
0:33:32and i mean hazing and amazing better sound and you was
0:33:37you always helped me in many ways you was very busy hiking on many projects
0:33:43but every time i had
0:33:46a question it drop by don't question or a sys admin question you was always
0:33:53available twelve and
0:33:56i will never forget in
0:33:59and
0:34:01i think i wanted to create a slight so we can remember em hold together
0:34:06or
0:34:09okay
0:34:10the should be able for me if you have any question
0:34:14it's just right well the thing is i if you have services that you do
0:34:19want you know big even discussing here it'd be nice i does one person wanted
0:34:26L X R four could searching so that's one
0:34:31one avenue there
0:34:33do you guys have other things that you think would be beneficial
0:34:41gonna
0:34:47a building windows and mac binary is
0:34:54on what exactly
0:34:57it stars or i guess i don't know
0:35:02i want
0:35:13i i'm here you can king use the
0:35:19use
0:35:29i was thinking about motion maintainers
0:35:41and that there's been some discussion about doing build windows binaries on the from build
0:35:47like that or
0:35:48we have a seem to set up to do the old will be used for
0:35:53a always project them or
0:35:55so i mean i think it's mostly question having somebody to maintain build scripts and
0:36:00that's for the across to automatically why do native bills have windows be and that's
0:36:04a trick your if you something would have to work all the are we are
0:36:08how to properly but it's only doing crosstalk with like a menu double you is
0:36:13certainly very feasible to really question
0:36:17for mac i'm
0:36:19mac is are difficult i think we'd have to have your some way of getting
0:36:22back centre infrastructure i think that it's
0:36:26it is these it is anything is feasible what it's a little bit more
0:36:31tricky
0:36:33so
0:36:39about a
0:36:44well i'm sorry
0:36:48what's that exactly you don't know victorious
0:36:52okay so i i'm sorry i don't actually we so people will only well if
0:36:59we're doing you have clone you don't know what it is exactly is the reason
0:37:05is the we closed are using pull requests
0:37:08we don't wanna do the requests and intimate i would looks you right just we
0:37:12don't wanna do that with the close or service that is one team and someone
0:37:17else but we like internally to be you really useful for contributions
0:37:24that definitely you to put it up but it will take a look at it
0:37:30so there is the that is in this but five back in two thousand nine
0:37:35problem at the comments and analyse it well there's like this to submit the transmission
0:37:40file
0:37:42and last number one and we'll come in the final after a you hang on
0:37:48that is that
0:37:56the main problem i see in setting up a guitar is first
0:38:01it's really our to stop really
0:38:05and manning sys admins told me that it's being an item our well so are
0:38:11we going away from our work grew around i mean we do review patches and
0:38:19sailor
0:38:20and will be spending a force in a pretty bond sale or does it make
0:38:24sense to you setting up on order
0:38:28it's tunnels which is it orders
0:38:32i mean about the per class
0:38:39so if it's not good have the projects i don't remember the name of you
0:38:45know what
0:38:47is that right i don't know how to discipline a because we really happy with
0:38:52it well the problem is that gets allowed i don't or is introduce a completely
0:38:57different way of making at else
0:39:00right and well if it's not feasible then we don't that no women is that
0:39:05women H everything propelled all and all the scripts we use our freedom to talk
0:39:12about work tools problem held up instance right so that mean providing all the scripts
0:39:19for another or another service like it's about which is completely different H else system
0:39:26right so it's not something that can we do if it's not really of them
0:39:32we better users as of or something else i guess the last to the other
0:39:36question i from my experience a reviewing pull requests this a lot
0:39:44use here
0:39:46and then i would have to you are the you percent that in something like
0:39:49could have or ataris is
0:39:53i guess it the they work flow is a lot more
0:39:57it's nice you can also i mean we really we have meter will be able
0:40:03to do like we compare comparing that the master runs with the future ranch
0:40:09but of course will relying on a on a also service so it's like we
0:40:14i don't think it's nice to have the me that i don't think that we
0:40:17shoe we want to ask people to use a aggressive so receive if we want
0:40:23to the things nicely
0:40:26so i don't know but maybe we could do is fill out a way to
0:40:29just percent that they have
0:40:31without setting up the whole infrastructure there was also proposal to so top review to
0:40:37review board
0:40:39but again sure is just a review board which is another review tell and then
0:40:45we have splinter which is you've used by but still and other non developers so
0:40:53that works role we got tired of i mean you send your patch a review
0:40:58board and you say some top out there you say that but the thing is
0:41:03that right now if you're working on a ranch it's already if you're using was
0:41:08that there have to generate the patches and then i will a hole in have
0:41:13some information with language and
0:41:17able to that but still a
0:41:20about so once that not having another one
0:41:25at the moment our but still a sense is completely separated from there is no
0:41:31interaction between it so you have to my only great departure submitted to but still
0:41:36and that's all we nice about and integration system between but still and i work
0:41:41at but that's also a bit harder because we use to get
0:41:47which is us off to return see that does
0:41:50our web interface craze this also on the goal of something like that that's we
0:41:56have some have to live in see right if we were a if someone wrote
0:42:01be able to expansion to do that maybe we can great on extension for but
0:42:07still okay want to propose
0:42:12i'm not suggesting the you do it up and then you can asks if someone's
0:42:16up to do it
0:42:19so we need more manpower wide come to develop new tools right now
0:42:25i just suggestion to the A B and they have to are well companies organiser
0:42:33it is nice to have a open source event manager to centralise
0:42:38to happen environment to or scalable one and the problems and but you but the
0:42:45troubles will probably you know this is more professional for programmer number us
0:42:52to be also well all you have
0:43:08if you're review paper submission and then out emails that they're too good during the
0:43:12process so that's how project if you guys have any suggestions any ideas for this
0:43:18you should look for disk are just too
0:43:22it was is a dog maybe you got you very tall because i i'm sure
0:43:27specific a reference the things you need for no way show right so
0:43:33i think that can we know it is definitely welcome
0:43:37you also should scroll to building or management system for conferences so
0:43:46just a lot of time
0:43:52what
0:43:58i know workshops
0:44:05it's not question i just really wanted to think you guys you've been doing and
0:44:09a really awesome job be i mean especially for collecting it he's been really helpful
0:44:14and i reassure all the other team's has the same just say sent thank you
0:44:17very much
0:44:18but