Wednesday, September 17, 2008

Making Evaluation Meaningful to Stakeholders




In 2002, Western Michigan University released a checklist meant to assist in making evaluation meaningful to education stakeholders. The checklist establishes three major areas of interest to make these evaluation meaningful to wide range of stakeholders: 1) Assessing the customer base, 2) Formatting the evaluation report, and 3) Disseminating the information and educating the stakeholders. Click the link to read through the checklist. Do you think this type of a checklist would be useful in social work agencies? Let me know what you think.


54 comments:

Stuart Jaffe said...

Josh, Outline strikes me as overdone and overlooks the reality that most programming of services occurs incrementally. In many instances, interested stakeholders including funders, regulators, boards, etc. may be directing an initiative. As shared in class, I also believe that program evaluation element should always be developed prior to initiation of a new program or service whenever practical.

S.Ray said...

I think the article makes some good points with the “checklist” to aid evaluators in regard to making the evaluation more meaningful to a diverse population. I think a similar checklist would be useful in social work agencies. It seems essential in assessing the “customer base” that acknowledgement should be made concerning the fact that professions have their own language and efforts must be made to connect with community partners. I think the point made in terms of disseminating the information and educating stakeholders is also a valid one. Programs need to consider the importance of the public’s perception of the program, because the perception will have an effect on securing funding sources.

scarlett holt said...

I do think that this type of checklist would be beneficial to social work agencies. I noticed several points made throughout the article that would have made things easier for me and my co-workers. There were obvious details that should be considered under each heading in the article, but some were food for thought.
I thought that it was a good idea for the steakholder's vested interest to be examined in that some of the aspects of the evaluation could be tailored around this. Also finding out the viewpoints of the opposition contrary to the research is important and can place the evaluator at an advantage. Going along with this, to address the viewpoints of critics in advance is valuable, in that it will eliminate some of the "tit for tat." Offering a cascading manner of an evaluating assists in not overwhelming the reader up front, and seems to get to the point of the matter, as does offering a bulleted summary of findings. I also liked the idea of offering a glossary of the terms in the report as it helps a general audience in understanding what is being said. Finally, to distribute the results in a variety of ways allows for additional outreach.
Scarlett H.

Anonymous said...

Conteria Williams, original posting

The stockholders check list, strikes me as being a little too perfect. Through reading the checklist, I had to be honest with myself and through daily experinces, I am aware that things don't always go according to plan. I feel that if before a program is put into place, that the different stockholders, who are involved need to throughly explore of all their available resources so that fewer clinches will occur when the program gets up and running.

Anonymous said...

Conteria Williams comment to Josh

I agree that program evaluation elements should be developed prior to initiation of a new program. Through doing things before hand, it will allow the stakeholders who are considering getting involved a complete picture rather than them going into a project blind sided.

jefN gadsden said...

I felt that she had some great suggestions, especially about breaking down the jargon and putting it into terminology that anyone can understand. I like her use of the phrase “packaging in a user-friendly manner.” Sometimes our evaluator will start “speaking in tongues” and we have to ask her to restate it in English.

jefN gadsden said...

I concur with s.ray about the fact that professions have their own language and social work is not exempt. It is always better to use understandable words than to promulgate your esoteric cogitations, or articulate your superficial sentimentalities and amicable, philosophical or psychological observations. Let your extemporaneous descantings and unpremeditated expatiations have intelligibility and veracious vivacity, without rodomontade or polysyllabic profundity.

Anonymous said...

I think that the checklist would be beneficial for social work. This example is pretty rigid, and I am not sure that an exact replica would fit in the social work world (since we must frequently adapt to the environment that we are placed in), however I think it could be a constructive outline for use in stakeholder related issues. I was impressed at the basic guidelines for getting the information out there to the education stakeholders. I also liked that in section two of the report they stressed to make the report understandable to all stakeholders. This would be an important factor in gaining support. Overall, I think it’s a good idea but needs more room for flexibility for it to work in the field of social work.
Heather M.

Anonymous said...

Re: S.Ray

In regard to Suzanne’s comment, I completely agree with your point in considering the public’s perception. I had not though about the fact that if the program is viewed negatively by the public that you would not have a lot of funders willing to “stick out their neck” so to speak. No one wants to be involved in something that makes them look bad, especially when finances are involved.
Heather M.

Tiffiney Brittingham said...

The checklist was very detailed. Social work agencies should have some type of checklist. However, I am sure many social work agencies have some form of a checklist. A clients assessment many be seen as a checklist for the client needs. Anytime a client seeks service, I would say they are stakeholders as well.

Tiffiney Brittingham

Tiffiney Brittingham said...

I agree with Corteria...the checklist was a little too perfect. Although we may want to have things in order and in place having every option covered, but it does not always go as plan or was we expect. The checklist is and can be useful, but never lose the touch to improvise.

Tiffiney Brittingham

Kristie Ninesling said...

I agree with Scarlett's comment regarding the Evaluation article. The check list would be beneficial to an agency as well as anyone attempting to make an evaluation meaningful to stakeholders. The idead of a glossary is both beneficial and informative to anyone invloved in the evaluation process. Also, knowing how your opponents will think and address the situation prior to the evaluation is helpful for anyone conducting an evaluation.

Kristie Ninesling said...

This article was extremely interesting and I agree with the three concepts proposed to making an evaluation more meaningful to stakeholders. The first concept "Assessing the Customer Base" (preevaluation), can be necessary and informative to the evaluation process, it allows you to know exactly who your customers are. The second concept "Formatting the Evaluation" (during and postevaluation)this seems interesting in that organizes the information making everything nice and neat for both evaluators and stakeholders. The third concept "Disseminating the Information and Educating the Stakeholders" (postevaluation), is basically the final product in the attempt to make an evaluation meaningful to stakeholders. Afterall, what good is an evaluation if it holds no meaning to the stakeholders? ... very good article. Kristie R.

Anonymous said...

I also think that it is important as a stakeholder to at the very least consider some of the issues brought up by the checklist and I agree that when possible, program evaluation should be a part of the initial program development process. I also think that while many of the things covered in the check list seem to be impractical, the idea of a checklist is important and can serve a good purpose - as someone stated, it would have been very helpful for them and their coworkers. I do however like the idea of creating a checklist as a way to assist in program evaluation and think that it is not something that should be created once the program is in place. I think that too often we as workers get caught up in our day to day work and our minds are more likely to tease a checklist without considering some of the important aspects of the program that were originally considered during program development. Stacy C.

Anonymous said...

As i read each comment posted regarding this topic, i agree with S.Ray the most. The article is a "checklist" to help aid the reader and educate the use of terminology. How can you disagree with information that is provided to educate? As needed, the article can be used as a useful tool. joann

Anonymous said...

I agree with S. Ray about the "checklist". I do feel that this similar type of list would be very beneficial to social work agencies. It is important to take a well rounded approach when working with clients and this list was structured and covered many areas. It is important to not leave a key piece out of the puzzle when working with a client.
Robin G.-Gadsden

Anonymous said...

In response to creating an evaluation presentation meaningful to all stakeholders, it might be practical to create two presentations, one for corporate and agency stakeholders with all of the appealing jargon that they understand and appreciate in addition to one for parents, community interest etc. that would be less complicated and more meaningful in terms of how does this directly affect me, my children, and my community regarding positive change.

LDW

Anonymous said...

I agree with Scarlett that social service agencies might benefit from a checklist to cover all areas of interest related with evaulation projects, although my concern would be the consumption of time.

LDW

Anonymous said...

original posting

One thing in particular that stood out to me while reading the stakeholder checklist article was that in the first line the authors underlined the word all. I just don’t know if it’s a good idea to say “making evaluation meaningful to all education stakeholders, it may give people the wrong impression. I just thought that by adding the word all it made me as a reader really look at the checklist a little differently. I think that having a checklist for stakeholders is a good idea, but I also think a checklist has to be re-evaluated and changed on a regular basis. However, sometimes having a checklist can make people a little lazy. They may have had an idea or a thought and because they used a checklist they disregard it, forget it, or just not mention it all together. If this happens then there may be some vital information missing somewhere. I know when I was a nursing school at DCH they used a checklist format for patient assessments. At first I thought I really liked it because it helped guide me and remember things that I sometimes might not have thought about. Then when I switched to St. Vincent’s where they still use a paper charging format, and you have to write all your notes. Using that format made me really focus on each specific patient and their problem areas rather than just going down a checklist and going through the motions. I am not saying that a checklist for education stakeholders isn’t a good idea, I am just pointing out that there are positives and negatives to both sides.

Sara S. (Tuscaloosa)

chadknight said...

The checklist outlined in this article would definitely be useful in guiding the evaluation of social service agencies. The checklist contains some elements of the EBP process, e.g., examine the relevant research related to your evaluation. The checklist states that the consumer base should be examined and that any reports should be made available to them and in language the consumer understands. This would be helpful to the clients of social service agencies, and would make the evaluations of those agencies more in line with the code of ethics of the profession. The point about including a one-page bulleted list of findings included in the executive summary is a good idea.

Anonymous said...

In response to Stu. J.

I agree with Stu. J.that a checklist needs to be developed prior to the implementation of a new program or service. As I stated in my original posting on this topic I also believe checklist that are already in place need to be re-evaluated on a regular basis. How often that occurs should be based on the program, service, stakeholders, and agency. I also agree that the checklist in this article seems to be overdone and may be overlooking something. I don’t think that checklists are a bad idea, but I do think they should be used in conjunction with other ideas to get a more holistic picture.

Sara S. (Tuscaloosa)

chadknight said...

I agree with Jeff about the "user-friendly language." Social workers learn to use the same language their clients use in order to build a better relationship. The same idea should be applied with dealing with stakeholders.

Anonymous said...

I would not say that every detail of this checklist would be applicable to a social service agency. However, I think that this checklist could serve as a model to use for developing a checklist for social service agencies. I think that the development of a checklist goes back to an agency making it clear what the mission, goals and objectives of the agency are. The checklist helps to guide this process and ensure that the agency is fulfilling its desire purpose. The clearer the checklist for the agency, the more funding and assistance the agency is likely to receive.

Anonymous said...

I agree with Stacy C's comment. Some of the items that I saw on the checklist would be a little time consuming. I understand how implementing visual data (ie. graphs) could help serve as a guide for an agency. This date could help to guide the program evaluation process by looking at the present moment and making inferences on where the agency would like to move towards. However, it might be a little difficult to get all the specifics for this information.

I also think that it is important that the program evaluation process should occur ahead of time. Based on my work experience in a social service agency, I know how fast paced and demanding can be. It is so easy to get caught up in the challenges that one is faced on a daily basis. As a result, one might lose site of the focus of the agency and not pay attention to the evaluation process. Also, it might be a good idea to have someone in the agency who is specifically employed for program evaluation purposes to help with the specifications.

And lastly, I agree with LDW’s comment. It would be more feasible to have two separate presentations for different stakeholders. However, I think that it would be cost-efficient if the agency has one presentation that can be understood by individuals of different educational backgrounds.

Unknown said...

I think the author made some really good points in the article. I particularly liked the way she broke down evaluations into three levels. Stakeholders should appericate such meticulous procedures in evaluating effectiveness.

Anonymous said...

I am enjoying learning about stakeholders. This article was very informative. I think that is was important for the author to mention that not only is it important to know who the stakeholders are but also what interests they have because they could be linked directly or indirectly to the project being evaluated.

I also thought it was important that the author mention to cite the critics’ potential counterpoints and support your findings with arguments based on empirical data. There is a difference between merely verbally arguing and presenting empirical data.

Ariel C.

Anonymous said...

Re: Stu

I believe that you are right about the fact that the program should already have an established program evaluation element, however i do think that it is important for those who do not have one to use this checklist as a guide.

Anonymous said...

In response to Jeff...
I agree that there were some great suggestions, especially about breaking down the jargon and putting it into terminology that anyone can understand. It is very important for us as social worker to implement “packaging in a user-friendly manner.” Sometimes our policy sort of start “speaking in tongues” and we have to ask for clarification, from the almighty Montgomery.

Virginia H. Tusc.

amanda said...

I think that the check list would benefit a social work agency, but I am a check list type of person. I like the way the outline breaks everything down in order to help others understand. I thought the way they listed the evaluation report would be great, simple and to the point. I think the checklist would help everyone involved in the evaluation to be on the same page.
Amanda H

amanda said...

I agree with Heather that we may need to make changes as needed in order to fit into our agency. I do not think that the same checklist would work at every agency.
Amanda H

Anonymous said...

Jeff, could you repeat that verbage? I comletely agree with Jeff and Chad. We need to communicate with client's by using understandable language. As social workers we have to modify speech depending on the population we are addressing or communicating with at the time.

Anonymous said...

I think the checklist would be useful in certain agencies. It was definitely useful in making the evaluation more adaptive to the different populations targeted for business and community partners as well as the educational community. Since accountability is the key issue then the educational partners need to have an appropriate evaluation design to connect them with community partners to keep them involved. From a hospital social workers standpoint, in my experience checklist were effective in keeping the focus and including relevant information but on the other hand if your assessment was hand-written it tended to include more pertinent information that is omitted from the checklist. Seemed to cover all the bases(plus some) rather than the standardized form.

Unknown said...

I agree with Destin in regards to accountability. As evaluators, we really are accountable to our stakeholders. I mean as cheesy as it sounds, they really are the only one's with something "at stake". Good point Destin ! :)

Anonymous said...

ORIGINAL
Ideally program evaluations should be developed prior to a programs inception. It should not be too rigid though. There needs to be room to address the unexpected as the program evolves. Determining who the audience is an important in the delivery of any service. In terms of looking at the stakeholders in education and Social Work, one must be careful to use a language understood by all as there are a diverse group of stakeholders. Our profession is similar to that of education in that we use buzzwords or acronyms in describing interventions and outcomes. When writing progress notes and other documents, we need to be mindful that the stakeholder may not have the same background as we do. We must document in a language that others can understand. When we as Social Workers address an interested party, we need to be mindful of both our written and verbal communication and speak in laymen terms so we do not exclude.
Matt g. (Gadsden)

Anonymous said...

REPLY
In reply to Sara S: When I did my internship on the psych unit at RMC, I noticed the nurses used a checklist format that you are talking about. There was also a narrative space as well. Where I work we use a similar format. I find this works well because we must address multiple objectives through out the course of daily documentation: symptom increase, affect, med compliance, and the list go on. If I were to rely on memory for this, I would surely leave something out. It also organizes it in a way that someone reviewing for quality control can quick scan the document and glean the important information. Sometimes a person does not fall into a category of a check box. That is where the narrative section comes in. This gives the clinician space to expound on the list or check box. You make a good point that if we are not careful, lists can encourage us to go through the motions and depersonalize each individual client.
Matt G. (Gadsden)

Karen P said...

I think components of this checklist would be quite useful within social work agencies and in other programs or entities that have evaluation components. Just as this checklist has been developed by Western Michigan University for education and noneducation stakeholders, social work agencies could adopt portions of this checklist and tailor it to meet the needs of their agency/program stakeholders. One sentence in the article that should be highlighted is: “evaluators need to recognize the current makeup of education stakeholders and format their (reports) to be meaningful and understandable to noneducation stakeholders in the community”. I would hope that this is a concept that transcends beyond the educational system, but is a concept held by most, if not all, evaluators. If reports are not meaningful and understandable to all that have a vested interest (direct or indirect) in the outcomes, worst case scenario, we endanger the continued existence of a agency and/or program by potentially losing the interest of key, influential stakeholders because they fail to see the impact and usefulness of the program/agency existence.
I also feel that the second component of the checklist (Formatting the Evaluation) appears to be one of the most integral parts of the checklist. In the context of social work agencies, for non-agency stakeholders, some of the key areas from this checklist that would be the most useful are the bar graph, pie charts, and other graphical displays; the one-page, bulleted summary of the evaluation findings; and a glossary of related methods and terms used in the report. These items are fairly simplistic and can definitely promote understanding of the information, thereby making it more meaningful.

Karen P said...

Response to Latasha T:

You make an excellent point about the clarity of the checklist and how it should be modeled to tie in with the mission, goal, and objectives of the agency. The purpose of a checklist is to simplify and promote understanding. It is a useful method and guide for all levels of stakeholders that can be utilized throughout the entire evaluation process. The more stakeholders can understand and relate to this evaluation process, the more likely they will invest their time, advice, expertise, resources, and even money (when appropriate) to help produce the best outcomes.

Gina Smith said...

Gina Smith/Gadsden - Response

Hey Jeff!!! I wanted to get out my dictionary and translate your post, but alas, my time ran out. However, it does remind one that though we may be Master level social workers one day and our vocabulary has been expanded by 100years of school, not everyone is on this level. Truthfully, I am not sure that I even understood all the eloquent words that you used. To be able to meet our clients at their level and to be able to ensure that we understand each other is more important than showing how well we learned our vocabulary words.

DC Bekkah said...

I liked this article. It seemed clear to someone (me!) who has had limited experience in effectively working with stakeholders. When reading the step about assessing the customer base, I thought about how our class discussed the power and interest blocks and it was more tangible for me. The format of the evaluation report seemed logical. The structure of the report seemed like it was a time saver because it aimed to meet different needs depending upon their interest. I can see how the last step is important, but I feel that it may be somewhat idealistic.

DC Bekkah said...

In response to Amanda H.

I am also a checklist person! I try to prioritize (steps 1 and 2). I know that I will never get everything on my list done (some of step 3). Realistically, I'm not sure exactly what I would be able to do to manage stakeholders while maintaining the other aspects of my job. However, I think that the article was a good introductory sample of how the author hangs on to her stakeholders.

T M R said...

I thought they made an excellent point about the language used in reports and evaluations being able to be understood by all involved parties. I recently had a meeting with a speech pathologist in a public school and she was talking about a child not "benchmarking." When I asked for clarificataion she looked at me like I was an idiot, but I explained to her that the common definition for benchmark was a baseline...In their school/field they have evidently made up an entirely new meaning for the word...Who knew?? We are often guilty of this same thing at DHR when we talk about ISPs, CIPs, GALs, and any other of a hundred different acronyms. We need to make sure we are aware of other's knowledge base when talking to them.

Tysie Baker said...

I thought that this article was very interesting because it clearly explains the importance of making everyone involved in the education reform at stakeholder. First, it outlined the three levels that formulated an organized checklist: 1) Assessing the Customer Base; 2) Formatting the Evaluation and 3) Disseminating the Information and Educating the Stakeholders. I also liked the fact that during the preevaluation, the evaluation, and the post evaluation stage information was thorough and precise as to how to properly conduct a program evaluation.

Tysie Baker said...

I agree with Kristi. It is imperative to include all stakeholders in the evaluation process at all times. Honestly, what good is evaluation if the key stakeholders are not involved or are familiar with what is occurring in the evaluation process of they are not included.

Anonymous said...

Walter said... the checklist is a guide for the evalution. What I am hearing is that the results are not understandable to all stakeholders. Evaluators language must be heard by everyone with a stake in the evuluation.

Walter L. agree with Kristie all the stakeholders should benefit and understand the evualuation report.

Teresa D. / Gadsden Center said...

I think this type of evaluation design and delivery method would be useful in some Social Work settings. For example, I work for a non-profit agency who receives funding from many sources, including members of the community. If an evaluation could be read and understood by them, they could be better informed as to the agency's efficiency and it's effectiveness. Who know's this could possibly provide more funding for the agency!

Teresa D. / Gadsden Center said...

Reply to jefn gadsden's post on September 21, 2008 3:03 PM

What did yout say?

My point exactly!

Anonymous said...

I think after reading over the check list that it would be very beneficial for an evaluation review. I personally like to have an organized way of going about conducting a review. I also believe having a list and knowing exactly what you are looking for can be very helpful and can cut down on time.

Anonymous said...

In response to Chad:
I also agree that social workers' should use terminology that applies to the group of people they are working with at that time. This should include their verbal and nonverbal body language. The checklist had some good points but may need to be personalized for each program.

T M R said...

Great point about the jargon, Jeff. Who knew you could use so many big words in one paragraph? I do think sometimes we get so wrapped up in the language though that we forget about the actual social work and the clients.

nikkig said...

This type of checklist could be beneficial to social work agencies, but she be tailored to fit the needs of indvidual agency.
Examining policy implications will be crucial to determining stakeholders involvment with the agency, and sustaining the programs longevity within in the community. However, in disseminating the information, I think social work agencies often overlook clients as key stakeholders. They tend to start from the top and somehow forget to get the information to those on the ground floor. These stakeholders are just as important to an agency and should be treated as such.

nikkig said...

Stu,
you make a good point. It can be beneficial to develop the program evaluation prior to the initiation of program services, but it is also key to continue internal evaluations which may require ongoing, incremental evaluations throughout the service year to continue to enhance services, and keep stakeholders aware of the change in direction the organization may be taking.

SWilliams said...

Amanda, I have to agree with you about the checklists for social service agencies. I feel that these agencies are overwhelmed many times with the amount of services that they provide and their consumer base. Sometimes these agencies only have a limited amount of personnel who can devote time to an evaluation and having a checklist would allow many more individuals to work on an evaluation, while maintaining the validity of the evaluation.ing

Debbie Walker said...

This article presented good information regarding the need to research before the research begins. The case was made for program evaluators to research and identify who the current stakeholders for the services are in the community. The evaluation report should be clear to educators and non-educators who may be stakeholders for the provided services evaluated.

Debbie Walker said...

I agree with Kristie that it is most important that an evaluation is user-friendly to the stakeholders of the program. Of course clinicians should be able to easily interpret the information offered in the report but all stakeholders involved should be able to acess review the information.