hello guys
is under very and grady leading the announces admin team
and this for very
is correctly
i notice a sub men
will be telling you about how we make up twenty min age
okay so
a bit will be telling you about how we men age our influx store
out of site softer side
what will be the future plans of the infrastructure
a lot of more stuff
so
i live at the word to three
for the first just lights
okay so what we do
so we met can so seventeen so we have a number of roles we have
been making infrastructure so that means number web services own about things like that of
course we have get
those kind of things
our main job to make sure everything is stable and you know and running smoothly
right so and we also
try to follow but every single team in the rubber in and copyright is to
have a problem so it something goes down or we hear about is the right
we are jobs trying to fix it
we also coordinated our hardware sponsors for
anything related you know for is red hat
we work with red hat and so forth
a big ever
was our sponsors i would doubt that about the next slide well our sponsors are
currently read a canonical down they are gonna state university
the open source lap
so do the right down in fact sort or has been really helpful with us
many times and i'm really thankful to them
and you can switch the
so
i would do we communicate with each other
we teach sys admin
we do have
a channel on that your see that's going on that are which is sys admin
we do have a public mailing list and we all set up private one where
we receive all the programs all the proper grounds
domain renewals
also we discuss and it in private
like private matters
and we also set up a lot of new services for better can users and
developers to know exactly what's happening when i went for accept or
and these services are
start does not than on that are and we have a we have a little
then we are
that's three will we show you
okay
so we have a good bunch of services
and indicators something is wrong
you connect to the service or you can even keep this page on your are
that's filtered
and you get each update when something goes wrong so let's say that something's wrong
with other
but we sure are red
that buttons saying that there is a problem and we have quality checking what's going
on there
so keep this page bookmark every time and before asking us on the sys admin
channel get yourself on this package and you know what we don't really know what's
going on
and
okay
then
we created
i'm not remaining liz i'm sorry for that but
when i'm sending my announcement made as i was sending them to desktop double is
a foundational list for mission announce so they created in fact that are announce whether
i'd be sending well my emails about what's going on only for accept or and
i also created a public not just you if you connect to ideas but the
number per week to username anonymous and plus or anonymous you see old the services
we run and the status of them so
it's even better than the start of the dog the them that or one because
not yours is a real time service so it sends a checks to the machine
if the czech was wrong it repulsed that if i there immediately so
just i don't have a preview of that but you can check that on your
or your machine stuff
okay
so
let's talk about the artwork
read a is kindly austin
a lot of machine for us
a total of twenty two machines
both vector machine and physical machines at the phoenix
on their things chewed at the centre
then canonical exhausting well two machines none of them exhausting the guy that website and
they're going on website
and down the oregon open-source lot exhausting
i wanna not be as box and
the testing machines for that website themed that and vested you about on in stock
we do not see lee use lee peered and capable
and
we actually have just eight physical machines which are but huge in terms of
a around C P U and so we use a lot of
we promotions
and there is something special and you now we name our servers
for example
demolition that currently all's a going on that are it's called socket
many other caulk on the box or create bar or something like that
and the name is are taken for day G T K we just
so if you will do a reverse D N S check on a rolls you'll
find out some strange names don't worry because is that is that actually expected
okay so let's go to the software
i will in age our web servers
we do have
an apache server in front of H a proxy
so we have
should bucks as which acts L's front ends
and the fact that can machines actually are
E P less so they don't have a public E P and there couldn't you
just accessible by this is something team
and i'll does H a proxy works there is round probably in the innocent the
N S entry so you're clean and will
randomly get an E P and connect to one of the two products is
and this is call fail over
so i wrote boxes are set up enough a little remote
and
we use an apache in front of a chip proxy because
we're ready back well the best
to ask ourselves so if you to check well than on the non services well
down are us
which ssl right now
and of a can machine are abused lee adjustable or only by a lot that
can also something
so time for free
okay so on the software side we have a number services or actually more accurate
way i'm kind of i'm going to describe what environment looks like so i'm not
so machines everything is N F S mounted so users generally have a consisted you
still are going well
not users but systems as ministers on the whole network we have open P N
for we she's posted outside the phoenix data centre for actually the private network so
you actually have to dpmc axis these then the phoenix data centre and that's just
right for security purposes we also have open L that
for our managing central access over the good this is primarily how we do accounts
i think it's not
in those people know what that is finally we have or are buying for serving
guinness
i don't think there's much to say about that we have to name servers well
to
cashed us requests and then finally we have a gonna jabber server which book talk
further and slides
the next that we have this prospects and the spend virus cover
to so jonas our mail server and of course we have some stuff to catch
spam
we germans were classes web platform for about that be that the minimum that gonna
that word and black second another
probably can sample are using more annoying
and that's a and we also have
okay nice extensions that can and other which is
so i per class with an that was about but the by jasper
and then we have you can that which is a nice note taking application
that let's you
it's text and data that whatever we been using extensively at the marketing team especially
and using it extensively for taking notes
for on like prayer meetings like honesty and so forth
and now i've got it yes
so i we manage ourselves
we evenly use pop but we do have an internal proper repository manage we the
get
and each comment we do is than notified to all this is augmenting so i
actually know was working on wall
and all the team is aware that they shouldn't archer specific model because i'm doing
some changes there
i'll pop i'm not sure i know many of you blocks leno our puppet work
no one
so top it is a centralised management system
so you keep your configuration files in one place
and that one place is called puppet master
you down synchronise you create model models
and you synchronise your machines to a specific model so you are able to recreate
an entire machine we just two clicks
and by running a specific common so it's really and they to rebuild an entire
virtual machine in the case for example you're operating from well five to wreck six
we also keep are standard operating procedures on our we key because
each softer women age is configured is i evilly customized so not this is something
is are aware about our we did a custom on is that we key or
like as a lot or adapt so this all is therefore do specific motivation you
check the soap and you have all distractions you need to find out are you
should probably men age a specific service that was set up by someone else
so we come to the backups are you we don't have a back lit a
machine
and we manage our backups in a strange way i mean
we were on a nightly us think around that sense well the doctor of all
the service to a north carolina know when
to an or carolina facility
and from there
we get all the backups on the tape on tape drives
and
when vote a little script
for that i was going to ask and exclude list that means that we can
exclude specific directories from that we of the for specifics five system and exclude
i dunno cache files for example or flies or been mounts so we don't duplicate
the data on the on the backup
well the lots of us sink are then sent to a machine well and that
machine generates i'm apart every day so we get the mail which does something like
well the machines are fine all the backups one fine or something is wrong with
the of with a specific machine so you should check what's going
and this is called a tear apart and you should think we should dangling providing
just very nice by transcript
so i want to we do our monitoring we are easily using now i guess
using not just with a lot of customisation
and the main one is called czech and K
second czech and K is a no additional software you can actually willing to not
guess
and you get
and ten or more of more checks
without doing anything special
it does that for it just work out the box you just up to link
it on when you're not just interface and configure a few things
we also have
i don't and those are really nice feature because
sometimes apache can go down
and
for whatever reason or you just got that time out i don't and those are
there for that
and our that are those they work in fact they
send a check to specific service
and if the czech is critical it automatically or starts they H T P D
demon
so they are very and a and then we have P N P for now
guess which is a C E mail are too
i don't know to comedy or
to collect D it creates a pair often service statistics over the time so we
know
a real i'm not then machines slowly is or how much ram is being used
during the during that time
so
the future plans which answer many of your are interesting
and many of you
are complaining about bags a lot
because our but still a sense is very old
is version three don't for one of the neural is far but for but why
that's great didn't come yet
the normal as many extensions
customise we customise of our but still a sense
with many extensions that the we have written by a lot of readers because i
don't know if
some of you know marine
and unfortunately is not contributing anymore and
why talking with william
the other day you offered in south to
as many or of writing
these extensions
to the new but still a format so we can probably planned an upgrade of
our but still are doing the next six months i guess it will probably take
that we will probably have to take down the service for around two days i
guess because the them
the database you which is around twenty gigabytes and restoring the world down will take
i wanted to six to forty eight hours i guess
the next future plans is
or a locating are machines at phoenix to currently our an actress so you switch
the networks which we are currently using is the one of another rack
and that's definitely something we have to fix and we are collaborating with the field
are guys to phoenix to
to have all the management consoles attached to the breach of the machines and so
we can track down
well the errors of the console of the physical machines
our doesn't work we thought each machine also management console part
you connect D is mush this us which this part to the natural speech
you get then E P i username and a password
then you got yourself into the V P N
and you cannot assess their web interface that allows you to report the mission check
the console rest are gonna should down the machine on song
and get all the data else about the method man a vector of for example
so we got all the details about the physical machine itself
we also plan to remove shots from master
because going to there are a lot or a lot of active accounts and
well of them do have S H adjust to master
and i filled is that this is not very good secreted speaking so will be
removing assess to master and we'll be creating
a script
to do that
for your locomotion and we have think about quitting hourly the web interface what you
cannot indicate using your old up a sort and you can just then click on
upload file you select your top are your trouble from your pc and you click
on in style
this we call death dp admin script from the machine and do the job for
you so we finally got rid of
of the shelves
and really proud of the last season
and thanks to about through is what did all the work
and will be monitoring the work it not the norm to talk to get out
and this without plan program i think applause festival of so when you comment to
the to the repository
it to we run a little nearer
that will saying well the content of your
of your model you have to get up
but will be announcing that is in a better way with more get those in
the near future i guess
which will the instructions you may need
it's just are coming or
it's
okay
you know
alright and the you want advantage is that we can track the activity around the
particular project
we just one click in we can see who's working what they're doing against master
right so
resistors and i i'm certain your
get more details later cool
well also given the fact that katie the are pretty bad this group shown i
don't know a few months ago and the they didn't have the backups
so we are also mirroring to make sure that we have to backups one sound
to the to the external facility every night and one get up
we did to time their guitar people because they've been really up for we does
and open a about or setting design
so actually
okay so this part is really the fun part this is the part where we
get tell you what you can use an and to help you to have cooperation
the first one is me but which is a it lets you do meetings on
I R C and what it nicely
it it's a it's a software that most people don't know about which is why
i'm discussing it here
we got that's you if you want you want to start you just say like
you tell be bopping gonna start the meeting and then you are able to have
all your discussions lark you can actually have somebody has a leader so for it's
really quite cool and
mostly done you can just kinda and the meeting and then you automatically get lots
published a on a web on the website now i don't have some complaints about
this because a lot of people want the logs to be private
and so that's what the feedback we've gotten and we are gonna look at how
to do be lot on a per channel
basis and so if we can be configured like say if you sort of meeting
on like how you know right you can expect that's gonna be public whereas if
we started one in marketing or you know some other channel then those would be
private
so that's one thing there
D this light has a link to be documentation so i will encourage you to
look through that use it and give us some feedback because i think it'll really
help us out when trying to collaboration
the next one is you think that a talk about this before
we use is a lot
and especially on the marketing team we did this
constantly internet quite like assembly said we for princes we were putting in but
we're doing summarise so each day and we've been using this it's
i basically you can have three four people adding this
this page and you can have chat and you also know who's type being white
because a lot different colours it
right we can always try to do a demo real quick we've had some feedback
on this as well
you know the one of the things that happen on this is that you only
get one either bad and if you have any us either pad before a lot
of people would like to use multiple because when you go to it you only
get this one page and if somebody's already using it then you don't really know
what to do it so you end up people and of going to pilot patterson
last so we're looking at trying to create
new ones on the fly so we don't do it before because we're worried about
spam but we will try to
doing experiments if it works and if you don't get spam right if we don't
get spam then well we'll figure out what to do at that point
and we had a whole session on this
well
but if you haven't been to if you didn't go to that chuck this is
a this is really cool because this allows you to do document storage right so
we document storage you can drag and drop whatever like open up this or any
number text sometimes if you have graphics design things like that it's a nice place
to put it
without having to use google doctor some other or other some other proprietary stuff so
the other cool thing is you can share that if you click on it and
say share with you can share with everyone you could create a public url just
like you can't are dark box and it's it works very well
so that's another nice thing
if you wanna use it
to create about and we'll be happy to give you access has using all that
much of the secure that
what about like to encourage them to use it more
so
are you okay
well then right need you put some screenshot so something like work in progress markup
of say website or whatever or just like a simple stupid disposable thing right which
store some services for but they are like i know how they are right so
what it used to do is to run this in a temporary folder on my
own server a but i'm very happy that it's a this thing excess because it
also i lost thank you very quickly easily charing public files winners and yes this
is what is looks like sorry so
thank you for sending data
and the guy course is relevant work an integration with you know right we'd gone
online docking online
services i want accounts are some reason i remember but reverie doing arrived are putting
like it's gonna show up are gonna documents and so for so i was just
wondering when it is you just going to be for national number for like the
Y O the foundation members only yet so this is sort of a something that
you get
for our as a as a
as a benefit so
we should we should definitely do that i do understand that it's probably us too
much from that if structure just to make it available to everybody but
is
i mean it's a service we expect you that you're gonna use it four gram
related activities as opposed to you know if it was why then you know we
would end up becoming a service provider right if it becomes too popular we might
you might have to
do something else because all that stuff takes place and you never know
the on the other it is to maybe have a
monthly payment yes
that would be a good a good a good go to
okay
not the
the main concern i have is about privacy because opening are instance to the world
we sure
provide some of the old documents saying
our paul is our legal places the S
so
okay
so i will like to take a few minutes to remember frown
a great friend
i work at we sat
for mass
i can tell to us actually on the for the way facts or and it
was
and i mean hazing and amazing better sound and you was
you always helped me in many ways you was very busy hiking on many projects
but every time i had
a question it drop by don't question or a sys admin question you was always
available twelve and
i will never forget in
and
i think i wanted to create a slight so we can remember em hold together
or
okay
the should be able for me if you have any question
it's just right well the thing is i if you have services that you do
want you know big even discussing here it'd be nice i does one person wanted
L X R four could searching so that's one
one avenue there
do you guys have other things that you think would be beneficial
gonna
a building windows and mac binary is
on what exactly
it stars or i guess i don't know
i want
i i'm here you can king use the
use
i was thinking about motion maintainers
and that there's been some discussion about doing build windows binaries on the from build
like that or
we have a seem to set up to do the old will be used for
a always project them or
so i mean i think it's mostly question having somebody to maintain build scripts and
that's for the across to automatically why do native bills have windows be and that's
a trick your if you something would have to work all the are we are
how to properly but it's only doing crosstalk with like a menu double you is
certainly very feasible to really question
for mac i'm
mac is are difficult i think we'd have to have your some way of getting
back centre infrastructure i think that it's
it is these it is anything is feasible what it's a little bit more
tricky
so
about a
well i'm sorry
what's that exactly you don't know victorious
okay so i i'm sorry i don't actually we so people will only well if
we're doing you have clone you don't know what it is exactly is the reason
is the we closed are using pull requests
we don't wanna do the requests and intimate i would looks you right just we
don't wanna do that with the close or service that is one team and someone
else but we like internally to be you really useful for contributions
that definitely you to put it up but it will take a look at it
so there is the that is in this but five back in two thousand nine
problem at the comments and analyse it well there's like this to submit the transmission
file
and last number one and we'll come in the final after a you hang on
that is that
the main problem i see in setting up a guitar is first
it's really our to stop really
and manning sys admins told me that it's being an item our well so are
we going away from our work grew around i mean we do review patches and
sailor
and will be spending a force in a pretty bond sale or does it make
sense to you setting up on order
it's tunnels which is it orders
i mean about the per class
so if it's not good have the projects i don't remember the name of you
know what
is that right i don't know how to discipline a because we really happy with
it well the problem is that gets allowed i don't or is introduce a completely
different way of making at else
right and well if it's not feasible then we don't that no women is that
women H everything propelled all and all the scripts we use our freedom to talk
about work tools problem held up instance right so that mean providing all the scripts
for another or another service like it's about which is completely different H else system
right so it's not something that can we do if it's not really of them
we better users as of or something else i guess the last to the other
question i from my experience a reviewing pull requests this a lot
use here
and then i would have to you are the you percent that in something like
could have or ataris is
i guess it the they work flow is a lot more
it's nice you can also i mean we really we have meter will be able
to do like we compare comparing that the master runs with the future ranch
but of course will relying on a on a also service so it's like we
i don't think it's nice to have the me that i don't think that we
shoe we want to ask people to use a aggressive so receive if we want
to the things nicely
so i don't know but maybe we could do is fill out a way to
just percent that they have
without setting up the whole infrastructure there was also proposal to so top review to
review board
but again sure is just a review board which is another review tell and then
we have splinter which is you've used by but still and other non developers so
that works role we got tired of i mean you send your patch a review
board and you say some top out there you say that but the thing is
that right now if you're working on a ranch it's already if you're using was
that there have to generate the patches and then i will a hole in have
some information with language and
able to that but still a
about so once that not having another one
at the moment our but still a sense is completely separated from there is no
interaction between it so you have to my only great departure submitted to but still
and that's all we nice about and integration system between but still and i work
at but that's also a bit harder because we use to get
which is us off to return see that does
our web interface craze this also on the goal of something like that that's we
have some have to live in see right if we were a if someone wrote
be able to expansion to do that maybe we can great on extension for but
still okay want to propose
i'm not suggesting the you do it up and then you can asks if someone's
up to do it
so we need more manpower wide come to develop new tools right now
i just suggestion to the A B and they have to are well companies organiser
it is nice to have a open source event manager to centralise
to happen environment to or scalable one and the problems and but you but the
troubles will probably you know this is more professional for programmer number us
to be also well all you have
if you're review paper submission and then out emails that they're too good during the
process so that's how project if you guys have any suggestions any ideas for this
you should look for disk are just too
it was is a dog maybe you got you very tall because i i'm sure
specific a reference the things you need for no way show right so
i think that can we know it is definitely welcome
you also should scroll to building or management system for conferences so
just a lot of time
what
i know workshops
it's not question i just really wanted to think you guys you've been doing and
a really awesome job be i mean especially for collecting it he's been really helpful
and i reassure all the other team's has the same just say sent thank you
very much
but