This list is intended to track privs by merit, rather than privs as recorded by the system. Please let me know if I missed anybody or got any wrong, which I'm sure I did. (I have blanket-decided that all admins are I2s in Web by merit if they weren't SHes before, since honestly, if any of you want to work on Web I'm not making you start out at screened. I mean, seriously, that would be screwed up. If any of you guys decide to get reviews, your next step will be merit I3.)
What is a memcache? Is it like a cache that tells you what X-man you are?
If you're not familiar with web caching in general, hit up the web-cache training post before reading this one. All set? Good.
Memcache is a system created by the LiveJournal development team to reduce load on their database servers. (It has since been adopted by other sites, but its primary users and maintainers are still affiliated with LiveJournal.) To explain what it's for, we're going to have to explain a little bit about how LJ (and lots of web apps) work. The way a page on LiveJournal is generally loaded is this:
Trust me, it's less complicated than it looks. The lines of this color represent an actual connection, the lines of this color represent paths the connection could have taken, and lines of this color represent the database's internal syncing mechanisms. So here's what's happening:
You enter a LiveJournal URL and hit return.
Your browser talks to the load balancer.
Unbeknownst to your browser, the load balancer goes out to find a web server to answer the request, and puts you in touch with that server.
For (mostly) unchanging content like images, this is as far as we need to go. The web server sends the content back to your browser. But most pages on LJ aren't unchanging; they're dynamic. So what happens next is that the web server goes to ask the database server to generate the content on it.
The database server digs out your journal entry (or any other dynamically-generated stuff) from its internal storage, and passes it to the web server, which then integrates it into the page and sends it back to you.
If you changed anything in the database when you hit the page (by updating your journal, for instance), the database server that you were talking to passes those changes to the other database servers. This process is called replication.
This picture gets more complicated when we talk about clusters and I'm elliding some subtleties about master/slave database relationships, but this is close enough for government work.
The problem with this system is that the connection to the database server is expensive, because that's where all the data really is and accessing it frequently involves talking to a bunch of disks. (When dealing with computers, disks are generally the slowest part of the system.) Ideally, as few connections to those servers as possible should happen. So what memcache does is store the results of your database query on one of the web servers when you make it, just like your browser cache does. Next time someone asks for that same piece of data, the web server figures out which server should be caching it, and checks there first before getting in touch with the database server. When you change that piece of data, the web server erases it from the cache when the database server is updated so that the next query will see the real data instead of whatever was there last time someone looked at the page. This cache is stored entirely in the memory of the machine (hence the name memcache), which makes it very fast.
This all sounds great, or at least it did once my head stopped hurting. So what's the problem?
There are a lot of things that can go wrong here, but there are two that are of primary concern to us:
Replag
Replag is short for "replication lag". This occurs when the database server that you sent your data to doesn't immediately update the other database servers about it, usually because it's a lot busier than it usually is for some reason. If this happens, you might connect to the wrong database server and see the old version of the data instead of the new one, and it's a piece of data that gets memcached, that old data might get put into the cache. If it's not memcached, this can result in someone seeing different versions of the page every time they reload it (since which web and database server you get sent to is, from your point of view, totally random.
Stale Memcache
If the memcached version of a page isn't cleared when the page is updated, the old version will still appear to be there (even if the new version has been properly stored in the database). This looks very much like a browser caching problem, except clearing the cache doesn't help with it.
So what can be done?
Most problems like this can be solved by time. And because they initially resemble caching problems, this means we'll never see them; we tell the user to clear their browser cache, and in the time between when we say this and when the user reads the answer, the blockage has been cleared and all is right with the world again. If the page gets updated by the user, that will also generally clear things out. Some problems of this kind are more persistant, though.
Web SHes have access to a tool that lets them clear the memcache for a particular user. In cases where a page is stuck in an old state, we're 100% sure that the page should be updated, and the old state is visible to the volunteers as well as the user (hence not browser cache), this tool can be used on the user that owns that page to force it to update. This is not a step that should be taken lightly, because it puts a lot of strain on LiveJournal's system to repopulate all that data, which is why use of it is restricted to SHes. However, if you're an interim, and you think a problem may be due to memcache, please IC and explain why you think this; just because you can't run the tool yourself doesn't mean you can't help out, and SHes aren't all-knowing. This tool also isn't perfect; it purges most things, but not every single conceivable memcached thing attached to a user. So if you don't notice any change in the user's journal after purging (and you've cleared your browser cache), it didn't do what you wanted.
Hi there! One of your friendly neighborhood web admins speaking.
We're going to be retooling the documentation in this community. Until we're done, it'll be locked down; I've removed most of the members, and set up membership as closed. The new docs will be posted friends-only so that they can be reviewed and commented on before becoming public information. The old docs in public posts will continue to be available until the new docs are ready to replace them, of course.
Please feel free to keep watching this community; once the documentation is ready, a public post will be made announcing that fact.
This document is intended to address the most common questions asked in web and how to go about answering/troubleshooting them. It shouldn't be viewed as a comprehensive list of all things that you'll ever see in web, but as a good starting point for common issues. Remember, in many cases, there isn't one specific 'correct' way to answer a request; it's just a matter of ruling out possible causes. Sometimes, you present the most likely answer, and offer the user a chance to regreen if that answer doesn't work for them. The elements of diagnosis and the guide to known issues, elsewhere in this community, will be of help.
Web-UI is a specialist category that deals with how users deal and communicate with LiveJournal's back-end (code, servers, and database). In particular, we deal with problems users are having with LiveJournal's code, servers, or database, and try to find resolutions to them. We do not, however, deal with questions concerning the normal use of LiveJournal's features (user education).
This document serves as a follow-up to Introductory Guide To Supporting Web-UI. It explains some of the changes that have occurred in the category since the Introductory Guide was written, and provides further clarification as to what the category entails.
In Review: What is Web-UI?
Web-UI is a specialist category that deals with how users deal and communicate with LiveJournal's back-end (code, servers, and database). In particular, we deal with problems users are having with LiveJournal's code, servers, or database, and try to find resolutions to them. We do not, however, deal with questions concerning the normal use of LiveJournal's features (user education).
What sort of requests fall under the scope of Web-UI?
The web category deals with three basic types of requests:
Something is not working as it should, and volunteers cannot reproduce it. -- This is generally caused by an error in the user's browser or computer settings.
Something is not working as it should, and volunteers can reproduce it. -- Issues of this type fall into three categories: bugs, features working as intended, and direct user error. It is important to note that only user reports of bugs fall under the scope of the Web-UI category. If something is working as intended (just not as the user expects), this falls under the scope of another category, as it requires only user education. If something is due to direct user error, such as incorrectly written overrides that cause their 'most recent entries' page not to display, this is again an issue of user education, and is not dealt with in Web-UI.
Temporary errors outside the control of LiveJournal support -- These are issues such as users not being able to reach the site or receiving error messages due to site overload. Problems with database clusters also fall under this category.
To give specific examples, the following is a short list of support requests that would be handled by the Web-UI team:
Issues involving the user's browser cache. 'I can't see new posts on my friends page, even though they've all posted!' or 'I can't make a new post to my own journal. I get the 'update successful' page even though I haven't updated yet!' are both examples of this.
Issues involving other browser features, such as JavaScript or cookies. 'I can't see the menu items in XColibur' and 'I'm getting an invalid cookies message, what does that mean?' are both questions of this type.
Issues involving errors in submitting data to LiveJournal. 'I can't comment on a friend's post, the form doesn't work right!' or 'I can only post short posts to my journal, why?!' are good examples of this.
Any issues involving the user having problems staying logged in. This problem may be due to their browser, computer, or network settings, and requires the specialist knowledge of the Web-UI team.
Issues involving web Unicode settings. 'Some of my posts show up with ???? instead of words, why is that?' and 'I'm getting a weird message about Bad Encoding, what does that mean?' are examples of this type of question.
Issues involving users unable to reach the site, or pages not loading as expected. 'I'm getting 404 not found errors!' and 'My friends page is redirecting to another user's journal page!' are both questions of this type.
Issues directly involving LiveJournal's back-end. 'My journal has been in read-only mode for three days!' is a cluster-related/site maintenance issue, and belongs in Web-UI.
Reports of bugs in LiveJournal's code. The key exception to this is any report of a bug that falls under the scope of another specialist category, such as UPI or Syndication. For example, 'my user picture isn't displaying right!' or 'The syn feed slashdot hasn't updated in days, why?' are not Web-UI issues.
How do we handle such requests?
Every request in Web-UI is treated as an individual case, as each user has his or her own browser, operating system, and settings. This might make it seem like each request in web is going to take a long time to properly investigate and answer, but in truth, many questions can be answered in a similar fashion, as either the issue is handled the same for most browser/operating system combinations, or the issue is site-wide.
Further reading on the process of diagnosis and answering in Web-UI is available here. One important update to the referenced document is the recent creation of the support_testers community, where volunteers can work as a team to attempt to research, duplicate, and even resolve bugs that have been reported in support requests. You do not need to have privs in Web-UI to participate - everyone is welcome.
What sort of requests do not fall under the scope of Web-UI? How do I tell the difference?
As previously mentioned, Web-UI does not handle requests that deal with educating users on using LiveJournal's features.
The easiest way to differentiate between Web-UI and another category is to analyse the question like so:
Is the user asking a 'how do I?' question, or are they reporting a problem? 'How do I' questions, such as 'How do I change the colors in my journal?' or 'I want to create a community but don't know how!', are handled in other support categories, not Web-UI.
If the user is reporting a problem with a feature, is the feature working as intended? If so, the question is a matter of user education, and is handled by other support categories. If, however, the feature is not working as intended, then it is likely a problem that falls under the scope of the Web-UI category.
There are some exceptions to the rules, however. The following cases are not handled by the Web-UI team:
Issues involving account validation errors. This falls under the category of 'user education', and is handled by the General/Unknown team.
Issues involving improperly written journal overrides or badly written styles that cause a journal page to display abnormally. Again, this falls under the category of 'user education' and is also handled by the General/Unknown team.
Issues involving errors in receiving or displaying LiveJournal emails, including any errors involving Unicode. Email problems fall outside of the scope of Web-UI, and are usually handled in General/Unknown.
Important Note: Volunteers in other categories who are working on requests that fall outside the scope of Web-UI, but may be aided by our expertise, should feel free to contact Web-UI's administrators for help. Requests should not be sent to Web-UI for diagnosis unless they are problems Web-UI handles.
Who administers Web-UI? How do I make contact?
Web-UI is currently administered by rahaeli and markf.
General questions concerning answering in Web-UI can be directed to learn_support or support_interim as appropriate. Correspondence concerning more private matters in the category should be directed to all admins at their @livejournal email addresses, or to support@.
What about reviews and mentoring?
Reviews are required for advancement in the Web-UI category. Directions for obtaining a review are available here. You may also wish to read leora's 'Guide To Reviews'.
At this time, Web-UI cannot offer a mentoring program, as there are not enough supporthelps to allow for it. In the future, we hope to offer such a program, but currently it is not possible.
If you would like to have more 'hands on' training in Web-UI, you can feel free to join us on the IRC channel, #lj_support (directions here). There you can ask questions to available Web-UI supporthelps. Please do note that the channel is meant to be an 'off duty' place, so sometimes no one will be available to answer questions, even if there are supporthelps present on channel.
Future Projects
The admins of the Web-UI team are currently working on further documentation to help volunteers working in the category. Specifically, a series of 'how to' type tutorials are in the works, which will help to show the thought process that goes into identifying, researching, and answering questions in Web-UI.
Requests for specific documentation or aid can be directed to the admins at their @livejournal email addresses.
Target Audience: Unprivs and I1s Summary: Covers in broad terms what a cache is, how browser caching works, why it can be a root or complicating issue in a presenting problem, ways of discerning these instances, treatment of caching problems, and norms for informing the user. Assumed Knowledge: Introductory Guide To Answering In Web-UI, Principles Of Diagnosis.
Introduction
Caching is one of the first problems a new volunteer will encounter in Web-UI. Newbies could be forgiven for thinking that almost every problem in Web-UI is cache-related, because caching answers are very prevalent. It is dangerous to assume that cache is the solution to every problem, but it is an important consideration when diagnosing any presenting problem in Web-UI.
What Is A Cache?
A cache is a directory set aside for temporary copies of the websites you visit. This is so that when you regularly visit a site, the content found there (especially images, which might otherwise take a while to load) can be loaded from your cache instead. This speeds up the performance of your browser. It can also save on bandwidth costs, not only for the user, but also for ISPs, gateway servers, and the originating server (in this case, LiveJournal).
Proxy servers also have caches. However, it is less common for this to be the cause of a cache-related problem, and it will almost always be necessary to exclude browser caching before moving into diagnosing proxy caching. So let's stick with browser caching for now. Proxy caching and other intricacies will be discussed in other documents in the future.
How Does Caching Work?
There are a few factors that determine whether a browser checks for a new version of a page, or loads an existing one from the cache. It depends to a large extent on browser settings. Many browsers only check a specific URL once per browser session, by default. In addition, some browsers display buggy caching behaviour on an ongoing basis. Patches and upgrades will sometimes help in this instance.
There are other factors as well. It can depend, for instance, on whether headers sent by the originating servers indicate an expiry time for the URL (related link). Proxy server behaviour can also be a factor. These complicating factors are beyond the scope of this introductory guide, and will be discussed in other documents in the future.
Why Is This A Problem?
Caching, itself, is not a bad thing, and for people on slow modems or ISP plans with download limits, it can be extremely helpful in managing internet usage and costs. Sometimes, however, browsers can display buggy behaviour, by persistently displaying a cached version of a page instead of loading fresh content.
There are two ways in which browser caching can relevant to presenting problems.
It can be a primary or root issue. That is, it can be the root cause of a presenting problem. One example of this is where a user has a friends page displaying old entries, but support volunteers cannot reproduce (ie, the friends page has fresh entries). In this instance, the user is most likely viewing an old copy of the page from an earlier time.
It can be a secondary or complicating issue. That is, caching can exacerbate a prior problem. One example of this is where the user experienced a transitory error in the past (for instance, some message to the effect that the server is not available) but continues to see the error even though the problem is resolved. In this instance, the user is most likely viewing an old copy of the error message.
Many thanks to timwi for the following documentation.
What are character sets?
Computers store information in bits. ("Bit" is short for "binary digit".) Each
bit is like a switch: It can take one of two values or states: "off" and "on",
a.k.a. "0" or "1". At some point in the history of computers, it was decided to
agree that 8 bits should make a byte; since each of the 8 bits in this byte can
be in one of the two states, there are two to the power of 8, or 256, possible
values for a byte.
Back in those days, it seemed intuitive to use one byte to represent one
character of text when stored on a digital medium. They assigned each of those
256 values a byte can have a meaning. Some of them were agreed to be control
characters -- for example, a line feed -- but most were printable characters
such as letters, numbers, or punctuation symbols. For example, the lower-case
letters a to z were assigned the bytes numbered 97 to 122. Thus, the following
message:
I am a fan of LiveJournal
was stored by a computer as the following series of numbers:
Computers gradually spread around the world, and a need arose to represent
more characters as other languages needed to be written. For example, German
uses the following special characters: ä, ö, ü, and ß.
So they started assigning more characters to more of those numbers and
eventually came up with an assignment that would cover most major European
languages. For example, ä was given the number 228.
However, 256 are not very many characters. Eventually, cultures with completely
different writing systems - for example, the Russians with their Cyrillic letters
- needed to represent their text as well. Since they, as they thought, would
never need to use German characters, they only kept the first 128 (which covered
English as well as all programming languages, mark-up languages and protocols)
and created their own assignment for the remaining 128. For example, д -
the Cyrillic letter De - was given the number 228.
With the advent of data-exchange technologies and in particular the Internet,
it soon became clear that these character-to-number assignments - called
character sets - need to be a agreed on. Furthermore, each document
(for example, a webpage) needed to specify what character set it was in; the
computer needed to know that, because if it did not, a German computer would
display a Russian text such as
Я твой друг
using the characters for the European languages, which ended up as an
unreadable mess:
ß òâîé äðóã
Notice how the д came out as an ä, because they both have the number 228.
What is Unicode?
Character sets have often resulted in problems and frustration, and they still
do today. Additionally, it was not possible to write a text containing both
German and Russian, because if the byte number 228 ever appeared in that text,
it was not clear whether that should be an ä or a д.
It soon became clear that this problem could not be solved without dropping the
constraint that every character would have to fit into one byte. Unicode
aims to solve the problem by providing several encodings that can be used to
represent text that contains characters from any language whatsoever. One of
those encodings, called the 8-bit Unicode Transfer Format or UTF-8, is designed
to use a single byte for the first 128 characters, while using several bytes
to represent all the other characters used worldwide. The first 128 characters are sufficient
for unadorned English text, programming, HTML, CSS and so on. Note, however, that even those writing in English will from time to time use additional characters, including smart quotes and accented letters for words and names originating in other languages.
Web-UI differs from other specialty categories in that it is a diagnostic unit. This means that the methodology of answering in Web-UI is different to that of other categories.
In other categories, the aim is to be comprehensive - to cover all likely explanations in one answer, and minimise the likelihood of repeated contact. In other categories, "Still Needs Help" is often considered a less desirable outcome. This is not the case in Web-UI. In Web-UI, it is necessary to take logical, methodical steps - to identify one or more probable causes, and diagnose by exclusion.
This is why clearing the cache, for instance, is such a common initial response to queries in Web. It will treat many transitory problems, and if the problem is more persistent, that will be confirmed when the user still needs help. Clearing the cache will not treat every error, but many basic errors will respond to it. The trick is learning to identify when it might help, and when it probably won't.
What Is Diagnosis?
Diagnosis is essentially the art of informed guesswork, on the basis of knowledge, experience, instinct, and probabilities. It involves taking the information about the problem provided by the user, your understanding of common problems and the processes that are used by the site generally, and learning to compare them. In time, you will learn to make connections between the two that will allow you to identify a group of likely causes for a problem, and then you can methodically narrow the list down until you can identify the cause and provide the user with a solution.
Articulating The Manifestation And Scope Of The Problem
Users usually do not know the right information to include in a request. For straightforward matters, this need not be a problem. However, for more unusual problems, you may need to query the user for more information. You will need to have at least a rough idea of where the problem lies in order to decide what, if any, detail is needed.
Some questions you might need to ask - either by querying the user or by research - include:
How long has the problem been going on?
Does the timeframe correspond to any user system changes? (eg, new ISP, browser upgrade)
Has the user attempted any corrective steps previously?
Can the problem be reproduced? (either consistently on the user's system, or by volunteers on other systems)
Have other users reported the same or associated problems, especially within a short timeframe?
You will also need to consider whether the user is mistaken in her or his information or interpretation of the problem. If the user is using technical words in context, or displays a methodical approach to resolving the problem, this is less likely. Use of technical words incorrectly, or use of imprecise language or l337, is an indicator that the user's information is less likely to be accurate.
Sometimes users will be unco-operative. Some have short attention spans. Some have problems with attention deficit disorders. Some have English as a second language. Some are just intimidated by technical language. Try to be patient. Make your requests progressively simpler, with short sentences or in point form. If you are having trouble extracting necessary information from a user, feel free to request the help of the Category Admin.
Very rarely, the user will lie. If you suspect that a user is actively lying (for instance, in the hope of being issued with a replacement account code), alert the Category Admin. No volunteer is expected to diagnose for an intentionally misleading user. The same applies to users who will not perform reasonable diagnostic tasks.
Avenues Of Enquiry And Comparison
You will be comparing this information with your own personal knowledge base. This includes your general computer knowledge, your general knowledge of LiveJournal, your knowledge of known issues, and your knowledge of the tools available to you.
Some avenues of enquiry you might need to consider:
Can the problem be attributed to user error or false expectations?
Do the symptoms, or the user's configuration, correspond to a known issue in LiveJournal?
Do the symptoms correspond to some other known issue? (eg, browser bugs)
Do the symptoms date from some relevant change in LiveJournal's code or systems administration? (See places like changelog, lj_maintenance, or news).
Could it be that the user is presenting with multiple problems, either separate or related?
More rarely, is there any geographic issue that may be causing or exacerbating problems with data transmission? (For instance, a marine disaster in the user's country could be interfering with offshore cables, or an earthquake could have knocked out major backbone servers).
These are all in addition to more generalised knowledge of browser and site behaviour.
Forming And Testing A Hypothesis
It is not necessary or helpful to create a list of every possible cause for a presenting problem. Some problems are so rare, you might only see them once in your whole time as a volunteer. Some will only crop up every few months. You don't need to knock yourself out accommodating these sorts of extreme possibilities until more likely probable causes have been excluded.
Try to use logic to pin down the general area of the problem. If a problem can be reproduced by many people, it's unlikely to be a problem with the user's browser or OS (unless all are using the same browser or OS, of course). Conversely, if the problem cannot be reproduced, it is very likely to be with the user's browser, TCP/IP settings, ISP's proxy or domain name servers, etc. Consider whether the problem could be a transitory problem that is now peristing because of caching. Form a short-list in your head of possible causes, and then narrow down. If you have two problems on your short list that would respond to clearing cache and cookies, then start there. If you suspect that a problem is browser-specific, you could ask the user if they can reproduce the problem in a different browser. This will assist in diagnosis and possibly allow them some other way of using the site in the meantime.
If you feel that this may only be the first step in the investigation, actively encourage the user to comment back if the problem persists. Believe it or not, many users actually don't want to be a bother.
Problems That May Arise
If the user either refuses to co-operate with diagnostic tasks, or claims to have undertaken them but appears not to have done so, bring the matter to the attention of the Category Administrator.
If the user becomes obnoxious or abusive, transfer the request to support@ and bring it to the attention of the Category Admin. No volunteer on the public board is expected to accept abuse in the course of their duties.
If you are investigating a problem that is rare or new, or if you are investigating with a view to further action (eg, Zilla reporting), tag it [ui]. This stands for Under Investigation. You may also tag [rfc] for Request For Comment if you have a freaky you would like other volunteers to test or provide input for. Zilla reports arising from support requests should be made by Supporthelps or the Category Administrator. Interims are not expected to make these reports.
Don't feel that you have to pursue a problem until it is diagnosed. If you feel out of your depth, you can ask for help, research it, or just walk away. You won't be perceived as having given a "wrong" answer just because your answer didn't identify or resolve the problem. If it was a reasonable line of enquiry on the basis of the symptoms and probable causes, then it was a right answer.
Written by: Deslea R. Judd Last updated: 17 January 2003
angrychicken
