hello guys

is under very and grady leading the announces admin team

and this for very

is correctly

i notice a sub men

will be telling you about how we make up twenty min age

okay so

a bit will be telling you about how we men age our influx store

out of site softer side

what will be the future plans of the infrastructure

a lot of more stuff


i live at the word to three

for the first just lights

okay so what we do

so we met can so seventeen so we have a number of roles we have

been making infrastructure so that means number web services own about things like that of

course we have get

those kind of things

our main job to make sure everything is stable and you know and running smoothly

right so and we also

try to follow but every single team in the rubber in and copyright is to

have a problem so it something goes down or we hear about is the right

we are jobs trying to fix it

we also coordinated our hardware sponsors for

anything related you know for is red hat

we work with red hat and so forth

a big ever

was our sponsors i would doubt that about the next slide well our sponsors are

currently read a canonical down they are gonna state university

the open source lap

so do the right down in fact sort or has been really helpful with us

many times and i'm really thankful to them

and you can switch the


i would do we communicate with each other

we teach sys admin

we do have

a channel on that your see that's going on that are which is sys admin

we do have a public mailing list and we all set up private one where

we receive all the programs all the proper grounds

domain renewals

also we discuss and it in private

like private matters

and we also set up a lot of new services for better can users and

developers to know exactly what's happening when i went for accept or

and these services are

start does not than on that are and we have a we have a little

then we are

that's three will we show you


so we have a good bunch of services

and indicators something is wrong

you connect to the service or you can even keep this page on your are

that's filtered

and you get each update when something goes wrong so let's say that something's wrong

with other

but we sure are red

that buttons saying that there is a problem and we have quality checking what's going

on there

so keep this page bookmark every time and before asking us on the sys admin

channel get yourself on this package and you know what we don't really know what's

going on




we created

i'm not remaining liz i'm sorry for that but

when i'm sending my announcement made as i was sending them to desktop double is

a foundational list for mission announce so they created in fact that are announce whether

i'd be sending well my emails about what's going on only for accept or and

i also created a public not just you if you connect to ideas but the

number per week to username anonymous and plus or anonymous you see old the services

we run and the status of them so

it's even better than the start of the dog the them that or one because

not yours is a real time service so it sends a checks to the machine

if the czech was wrong it repulsed that if i there immediately so

just i don't have a preview of that but you can check that on your

or your machine stuff



let's talk about the artwork

read a is kindly austin

a lot of machine for us

a total of twenty two machines

both vector machine and physical machines at the phoenix

on their things chewed at the centre

then canonical exhausting well two machines none of them exhausting the guy that website and

they're going on website

and down the oregon open-source lot exhausting

i wanna not be as box and

the testing machines for that website themed that and vested you about on in stock

we do not see lee use lee peered and capable


we actually have just eight physical machines which are but huge in terms of

a around C P U and so we use a lot of

we promotions

and there is something special and you now we name our servers

for example

demolition that currently all's a going on that are it's called socket

many other caulk on the box or create bar or something like that

and the name is are taken for day G T K we just

so if you will do a reverse D N S check on a rolls you'll

find out some strange names don't worry because is that is that actually expected

okay so let's go to the software

i will in age our web servers

we do have

an apache server in front of H a proxy

so we have

should bucks as which acts L's front ends

and the fact that can machines actually are

E P less so they don't have a public E P and there couldn't you

just accessible by this is something team

and i'll does H a proxy works there is round probably in the innocent the

N S entry so you're clean and will

randomly get an E P and connect to one of the two products is

and this is call fail over

so i wrote boxes are set up enough a little remote


we use an apache in front of a chip proxy because

we're ready back well the best

to ask ourselves so if you to check well than on the non services well

down are us

which ssl right now

and of a can machine are abused lee adjustable or only by a lot that

can also something

so time for free

okay so on the software side we have a number services or actually more accurate

way i'm kind of i'm going to describe what environment looks like so i'm not

so machines everything is N F S mounted so users generally have a consisted you

still are going well

not users but systems as ministers on the whole network we have open P N

for we she's posted outside the phoenix data centre for actually the private network so

you actually have to dpmc axis these then the phoenix data centre and that's just

right for security purposes we also have open L that

for our managing central access over the good this is primarily how we do accounts

i think it's not

in those people know what that is finally we have or are buying for serving


i don't think there's much to say about that we have to name servers well


cashed us requests and then finally we have a gonna jabber server which book talk

further and slides

the next that we have this prospects and the spend virus cover

to so jonas our mail server and of course we have some stuff to catch


we germans were classes web platform for about that be that the minimum that gonna

that word and black second another

probably can sample are using more annoying

and that's a and we also have

okay nice extensions that can and other which is

so i per class with an that was about but the by jasper

and then we have you can that which is a nice note taking application

that let's you

it's text and data that whatever we been using extensively at the marketing team especially

and using it extensively for taking notes

for on like prayer meetings like honesty and so forth

and now i've got it yes

so i we manage ourselves

we evenly use pop but we do have an internal proper repository manage we the


and each comment we do is than notified to all this is augmenting so i

actually know was working on wall

and all the team is aware that they shouldn't archer specific model because i'm doing

some changes there

i'll pop i'm not sure i know many of you blocks leno our puppet work

no one

so top it is a centralised management system

so you keep your configuration files in one place

and that one place is called puppet master

you down synchronise you create model models

and you synchronise your machines to a specific model so you are able to recreate

an entire machine we just two clicks

and by running a specific common so it's really and they to rebuild an entire

virtual machine in the case for example you're operating from well five to wreck six

we also keep are standard operating procedures on our we key because

each softer women age is configured is i evilly customized so not this is something

is are aware about our we did a custom on is that we key or

like as a lot or adapt so this all is therefore do specific motivation you

check the soap and you have all distractions you need to find out are you

should probably men age a specific service that was set up by someone else

so we come to the backups are you we don't have a back lit a


and we manage our backups in a strange way i mean

we were on a nightly us think around that sense well the doctor of all

the service to a north carolina know when

to an or carolina facility

and from there

we get all the backups on the tape on tape drives


when vote a little script

for that i was going to ask and exclude list that means that we can

exclude specific directories from that we of the for specifics five system and exclude

i dunno cache files for example or flies or been mounts so we don't duplicate

the data on the on the backup

well the lots of us sink are then sent to a machine well and that

machine generates i'm apart every day so we get the mail which does something like

well the machines are fine all the backups one fine or something is wrong with

the of with a specific machine so you should check what's going

and this is called a tear apart and you should think we should dangling providing

just very nice by transcript

so i want to we do our monitoring we are easily using now i guess

using not just with a lot of customisation

and the main one is called czech and K

second czech and K is a no additional software you can actually willing to not


and you get

and ten or more of more checks

without doing anything special

it does that for it just work out the box you just up to link

it on when you're not just interface and configure a few things

we also have

i don't and those are really nice feature because

sometimes apache can go down


for whatever reason or you just got that time out i don't and those are

there for that

and our that are those they work in fact they

send a check to specific service

and if the czech is critical it automatically or starts they H T P D


so they are very and a and then we have P N P for now

guess which is a C E mail are too

i don't know to comedy or

to collect D it creates a pair often service statistics over the time so we


a real i'm not then machines slowly is or how much ram is being used

during the during that time


the future plans which answer many of your are interesting

and many of you

are complaining about bags a lot

because our but still a sense is very old

is version three don't for one of the neural is far but for but why

that's great didn't come yet

the normal as many extensions

customise we customise of our but still a sense

with many extensions that the we have written by a lot of readers because i

don't know if

some of you know marine

and unfortunately is not contributing anymore and

why talking with william

the other day you offered in south to

as many or of writing

these extensions

to the new but still a format so we can probably planned an upgrade of

our but still are doing the next six months i guess it will probably take

that we will probably have to take down the service for around two days i

guess because the them

the database you which is around twenty gigabytes and restoring the world down will take

i wanted to six to forty eight hours i guess

the next future plans is

or a locating are machines at phoenix to currently our an actress so you switch

the networks which we are currently using is the one of another rack

and that's definitely something we have to fix and we are collaborating with the field

are guys to phoenix to

to have all the management consoles attached to the breach of the machines and so

we can track down

well the errors of the console of the physical machines

our doesn't work we thought each machine also management console part

you connect D is mush this us which this part to the natural speech

you get then E P i username and a password

then you got yourself into the V P N

and you cannot assess their web interface that allows you to report the mission check

the console rest are gonna should down the machine on song

and get all the data else about the method man a vector of for example

so we got all the details about the physical machine itself

we also plan to remove shots from master

because going to there are a lot or a lot of active accounts and

well of them do have S H adjust to master

and i filled is that this is not very good secreted speaking so will be

removing assess to master and we'll be creating

a script

to do that

for your locomotion and we have think about quitting hourly the web interface what you

cannot indicate using your old up a sort and you can just then click on

upload file you select your top are your trouble from your pc and you click

on in style

this we call death dp admin script from the machine and do the job for

you so we finally got rid of

of the shelves

and really proud of the last season

and thanks to about through is what did all the work

and will be monitoring the work it not the norm to talk to get out

and this without plan program i think applause festival of so when you comment to

the to the repository

it to we run a little nearer

that will saying well the content of your

of your model you have to get up

but will be announcing that is in a better way with more get those in

the near future i guess

which will the instructions you may need

it's just are coming or



you know

alright and the you want advantage is that we can track the activity around the

particular project

we just one click in we can see who's working what they're doing against master

right so

resistors and i i'm certain your

get more details later cool

well also given the fact that katie the are pretty bad this group shown i

don't know a few months ago and the they didn't have the backups

so we are also mirroring to make sure that we have to backups one sound

to the to the external facility every night and one get up

we did to time their guitar people because they've been really up for we does

and open a about or setting design

so actually

okay so this part is really the fun part this is the part where we

get tell you what you can use an and to help you to have cooperation

the first one is me but which is a it lets you do meetings on

I R C and what it nicely

it it's a it's a software that most people don't know about which is why

i'm discussing it here

we got that's you if you want you want to start you just say like

you tell be bopping gonna start the meeting and then you are able to have

all your discussions lark you can actually have somebody has a leader so for it's

really quite cool and

mostly done you can just kinda and the meeting and then you automatically get lots

published a on a web on the website now i don't have some complaints about

this because a lot of people want the logs to be private

and so that's what the feedback we've gotten and we are gonna look at how

to do be lot on a per channel

basis and so if we can be configured like say if you sort of meeting

on like how you know right you can expect that's gonna be public whereas if

we started one in marketing or you know some other channel then those would be


so that's one thing there

D this light has a link to be documentation so i will encourage you to

look through that use it and give us some feedback because i think it'll really

help us out when trying to collaboration

the next one is you think that a talk about this before

we use is a lot

and especially on the marketing team we did this

constantly internet quite like assembly said we for princes we were putting in but

we're doing summarise so each day and we've been using this it's

i basically you can have three four people adding this

this page and you can have chat and you also know who's type being white

because a lot different colours it

right we can always try to do a demo real quick we've had some feedback

on this as well

you know the one of the things that happen on this is that you only

get one either bad and if you have any us either pad before a lot

of people would like to use multiple because when you go to it you only

get this one page and if somebody's already using it then you don't really know

what to do it so you end up people and of going to pilot patterson

last so we're looking at trying to create

new ones on the fly so we don't do it before because we're worried about

spam but we will try to

doing experiments if it works and if you don't get spam right if we don't

get spam then well we'll figure out what to do at that point

and we had a whole session on this


but if you haven't been to if you didn't go to that chuck this is

a this is really cool because this allows you to do document storage right so

we document storage you can drag and drop whatever like open up this or any

number text sometimes if you have graphics design things like that it's a nice place

to put it

without having to use google doctor some other or other some other proprietary stuff so

the other cool thing is you can share that if you click on it and

say share with you can share with everyone you could create a public url just

like you can't are dark box and it's it works very well

so that's another nice thing

if you wanna use it

to create about and we'll be happy to give you access has using all that

much of the secure that

what about like to encourage them to use it more


are you okay

well then right need you put some screenshot so something like work in progress markup

of say website or whatever or just like a simple stupid disposable thing right which

store some services for but they are like i know how they are right so

what it used to do is to run this in a temporary folder on my

own server a but i'm very happy that it's a this thing excess because it

also i lost thank you very quickly easily charing public files winners and yes this

is what is looks like sorry so

thank you for sending data

and the guy course is relevant work an integration with you know right we'd gone

online docking online

services i want accounts are some reason i remember but reverie doing arrived are putting

like it's gonna show up are gonna documents and so for so i was just

wondering when it is you just going to be for national number for like the

Y O the foundation members only yet so this is sort of a something that

you get

for our as a as a

as a benefit so

we should we should definitely do that i do understand that it's probably us too

much from that if structure just to make it available to everybody but


i mean it's a service we expect you that you're gonna use it four gram

related activities as opposed to you know if it was why then you know we

would end up becoming a service provider right if it becomes too popular we might

you might have to

do something else because all that stuff takes place and you never know

the on the other it is to maybe have a

monthly payment yes

that would be a good a good a good go to


not the

the main concern i have is about privacy because opening are instance to the world

we sure

provide some of the old documents saying

our paul is our legal places the S



so i will like to take a few minutes to remember frown

a great friend

i work at we sat

for mass

i can tell to us actually on the for the way facts or and it


and i mean hazing and amazing better sound and you was

you always helped me in many ways you was very busy hiking on many projects

but every time i had

a question it drop by don't question or a sys admin question you was always

available twelve and

i will never forget in


i think i wanted to create a slight so we can remember em hold together



the should be able for me if you have any question

it's just right well the thing is i if you have services that you do

want you know big even discussing here it'd be nice i does one person wanted

L X R four could searching so that's one

one avenue there

do you guys have other things that you think would be beneficial


a building windows and mac binary is

on what exactly

it stars or i guess i don't know

i want

i i'm here you can king use the


i was thinking about motion maintainers

and that there's been some discussion about doing build windows binaries on the from build

like that or

we have a seem to set up to do the old will be used for

a always project them or

so i mean i think it's mostly question having somebody to maintain build scripts and

that's for the across to automatically why do native bills have windows be and that's

a trick your if you something would have to work all the are we are

how to properly but it's only doing crosstalk with like a menu double you is

certainly very feasible to really question

for mac i'm

mac is are difficult i think we'd have to have your some way of getting

back centre infrastructure i think that it's

it is these it is anything is feasible what it's a little bit more



about a

well i'm sorry

what's that exactly you don't know victorious

okay so i i'm sorry i don't actually we so people will only well if

we're doing you have clone you don't know what it is exactly is the reason

is the we closed are using pull requests

we don't wanna do the requests and intimate i would looks you right just we

don't wanna do that with the close or service that is one team and someone

else but we like internally to be you really useful for contributions

that definitely you to put it up but it will take a look at it

so there is the that is in this but five back in two thousand nine

problem at the comments and analyse it well there's like this to submit the transmission


and last number one and we'll come in the final after a you hang on

that is that

the main problem i see in setting up a guitar is first

it's really our to stop really

and manning sys admins told me that it's being an item our well so are

we going away from our work grew around i mean we do review patches and


and will be spending a force in a pretty bond sale or does it make

sense to you setting up on order

it's tunnels which is it orders

i mean about the per class

so if it's not good have the projects i don't remember the name of you

know what

is that right i don't know how to discipline a because we really happy with

it well the problem is that gets allowed i don't or is introduce a completely

different way of making at else

right and well if it's not feasible then we don't that no women is that

women H everything propelled all and all the scripts we use our freedom to talk

about work tools problem held up instance right so that mean providing all the scripts

for another or another service like it's about which is completely different H else system

right so it's not something that can we do if it's not really of them

we better users as of or something else i guess the last to the other

question i from my experience a reviewing pull requests this a lot

use here

and then i would have to you are the you percent that in something like

could have or ataris is

i guess it the they work flow is a lot more

it's nice you can also i mean we really we have meter will be able

to do like we compare comparing that the master runs with the future ranch

but of course will relying on a on a also service so it's like we

i don't think it's nice to have the me that i don't think that we

shoe we want to ask people to use a aggressive so receive if we want

to the things nicely

so i don't know but maybe we could do is fill out a way to

just percent that they have

without setting up the whole infrastructure there was also proposal to so top review to

review board

but again sure is just a review board which is another review tell and then

we have splinter which is you've used by but still and other non developers so

that works role we got tired of i mean you send your patch a review

board and you say some top out there you say that but the thing is

that right now if you're working on a ranch it's already if you're using was

that there have to generate the patches and then i will a hole in have

some information with language and

able to that but still a

about so once that not having another one

at the moment our but still a sense is completely separated from there is no

interaction between it so you have to my only great departure submitted to but still

and that's all we nice about and integration system between but still and i work

at but that's also a bit harder because we use to get

which is us off to return see that does

our web interface craze this also on the goal of something like that that's we

have some have to live in see right if we were a if someone wrote

be able to expansion to do that maybe we can great on extension for but

still okay want to propose

i'm not suggesting the you do it up and then you can asks if someone's

up to do it

so we need more manpower wide come to develop new tools right now

i just suggestion to the A B and they have to are well companies organiser

it is nice to have a open source event manager to centralise

to happen environment to or scalable one and the problems and but you but the

troubles will probably you know this is more professional for programmer number us

to be also well all you have

if you're review paper submission and then out emails that they're too good during the

process so that's how project if you guys have any suggestions any ideas for this

you should look for disk are just too

it was is a dog maybe you got you very tall because i i'm sure

specific a reference the things you need for no way show right so

i think that can we know it is definitely welcome

you also should scroll to building or management system for conferences so

just a lot of time


i know workshops

it's not question i just really wanted to think you guys you've been doing and

a really awesome job be i mean especially for collecting it he's been really helpful

and i reassure all the other team's has the same just say sent thank you

very much